Package: slapos.toolbox
slapos.toolboxSlapOS toolbox contains :HTTP Cache CheckerCrawl a website and check the HTTP configuration.Runpython2 cachechecker.py -o result.txt cachechecker.confExample of configuration::[web_checker]
url = http://example.org

working_directory = /tmp/cachechecker

# debug_level = info
debug_level = debug
file_log_path = /tmp/cachechecker.log

# http://labs.omniti.com/people/mark/ats_sa/slides.html#slide-18
traffic_server_cache = cHs
  cSsNfU

[header_list]
Last-Modified = True
Expires = True
Vary = Cookie,Authorization,Accept-Encoding
Cache-Control = max-age=600, stale-while-revalidate=360000, public

[no_header url=.*]
Set-Cookie = None

[no_header content-type=(image/.*|application/vnd.ms-fontobject|application/x-font-woff|font/truetype)]
Vary = None

[header content-type=image/.*]
Cache-Control = max-age=3600, stale-if-error=360000, stale-while-revalidate=360000, public

[header content-type=(text/css|text/plain|application/x-javascript|.*/javascript|application/vnd.ms-fontobject|application/x-font-woff|font/truetype)]
Vary = Accept-Encoding
Cache-Control = max-age=3600, stale-if-error=360000, stale-while-revalidate=360000, public

# Captcha
[header url=(.*/contactus$)]
Cache-Control = max-age=0, no-cache

# # IE PDF Bug
[header url=.*format=pdf]
Cache-Control = Private

[header url=.*format=]
Vary = Cookie,Authorization,Accept-Encoding
Cache-Control = max-age=600, stale-while-revalidate=360000, public

[erp5_extension_list]
prohibited_folder_name_list = web_page_module
  document_module
  web_site_module
prohibited_file_name_list = WebSection_viewAsWeb
  Base_viewHistory
  Base_download
  listnetworkbenchonetimeuploadonetimeupload is an HTTP application which only accepts one file to be
uploaded.0.142 (2024-02-15)promise/plugin: fix missing expiration date suport in whois check incheck_surykatka_json0.141 (2024-02-15)promise/plugin: implement whois check incheck_surykatka_jsonpromise/plugin: skip all promises when testing=True incheck_lopcomm_*0.140 (2023-11-30)promise/plugin: teachcheck_sdr_busy,check_rx_saturatedandcheck_cpri_lockto handle multiple radio unitspromise/plugin: adjust Amarisoft-related promises and tests to treatdatain*.json.logas real JSONpromise/plugin: adjustcheck_rx_saturatedmessages to provide details about maximum detected signal levelall: general improvement on code quality here and there0.139 (2023-09-22)check_url_available: new allow-redirects option0.138 (2023-09-21)promise/plugin: add PA over output power for Lopcomm RU0.137 (2023-06-15)promise/plugin: add alarms for lopcomm RU0.136 (2023-05-16)setup.py: Define description content type0.135 (2023-05-16)This release drops support for python2. Software releases using python2
are pinned to 0.128, the last version with tests passing on python2promise/plugin: add enabled-sense-list feature in check_surykatka_json0.134 (2023-03-09)promise/plugin: allow to check only certificate in check_certificate0.133 (2023-03-09)promise/plugin: improve check_surykatka_json assertions0.132 (2023-03-07)promise/plugin: fix check_websocket_available promise0.131 (2023-03-01)promise/plugin: add check_websocket_available promiserunner: totally remove runner directory (webrunner is replaced by Theia)0.130 (2023-02-08)promise/plugin: added alarms for ORS monitoring: check_amarisoft_stats_log, check_baseband_latency, check_cpri_lock, check_interface_up, check_lopcomm_lof, check_lopcomm_rssi, check_lopcomm_vswr, check_rx_saturated, check_sdr_busypromise/plugin: reworked methods in util.py0.129 (2023-01-18)promise/plugin: added alarms for node monitoring: check_ram_usage, check_network_errors_packets, check_network_transit, check_cpu_temperaturemonitor: use gmtime to query data in collect dbresiliencytestsuite,kvm: python3 fixes0.128 (2022-06-30)notifier: support Python 3.8+0.127 (2022-06-05)qemuqmpclient: support qemu 7+0.126 (2021-10-04)plugin/check_file_state: fail on first problem0.125 (2021-09-30)plugin/check_file_state: fix empty url parameter usageplugin/check_file_state: allow to assert absence of the file0.124 (2021-08-25)pubsub: Fix Python3 incompatibilities0.123 (2021-07-23)plugin/check_url_available: many improvements, now supports basic auth
WARNING, backward incompatible changes:
+ http_code option is now http-code
+ check-secure option has been removed. Users should specify http-code = 401 instead.0.122 (2021-06-29)promise/plugin: Fix check_socket_listening promise0.121 (2021-06-24)qemuqmpclient: Support qemu 5.x+0.120 (2021-06-23)promise/plugin: Add check_socket_listening promise0.119 (2021-04-21)runner: python3 fixes for the web interface0.118 (2021-03-29)dnsresolver: new tool for massive DNS queries0.117 (2021-02-03)promise/plugin: Implement check_service_state promise plugin0.116 (2021-02-01)runner: import werkzeug.middleware.proxy_fix.ProxyFix from its new location (for compatibility with werkzeug 1.0.x)0.115 (2021-01-27)check_surykatka_json: minimise messages to avoid information bloat0.114 (2021-01-22)apachedex: expect apachedex options to be passed as a filecheck_free_disk_space: Btrfs has no fixed inode limit0.113 (2020-11-11)apachedex: fix argument parsing0.112 (2020-10-20)test: fix for random failuresrunner: fix deployment of instance through GUI0.111 (2020-09-14)promise: Support http-header-dict in check_surykatka_json0.110 (2020-08-25)promise: Fix argument parsing in check_slow_queries_digest_resultpromise: Make check_slow_queries_digest_result use .xz compressed reports, which introduces a dependency tolzma.backports0.109 (2020-03-12)promise: Fix certificate expiration reporting in check_surykatka_json0.108 (2020-03-02)promise: check_certificate promise introduced0.107 (2020-02-28)runner: more Py3 support.0.106 (2020-02-14)promise: Drop UTC-now in check_surykatka_json0.105 (2020-02-12)promise: Support total_seconds in check_surykatka_jsonpromise: Support failure-amount in check_surykatka_jsonpromise: Bugfixes for check_surykatka_json0.104 (2020-01-09)monitor: Cleanup stale history.json files0.103 (2020-01-02)promise: Fix check_surykatka_json for http-only checks0.102 (2019-12-31)promise: Support SSL checks in check_surykatka_json0.101 (2019-12-12)monitor: the monitor.global.json is under /public directory not /share/publicpromise: Add check_surykatka_json promise0.100 (2019-12-10)monitor: stabilise bootstrap status file creation0.99 (2019-12-05)monitor: support configuration w/o legacy-promise-folderpromise: consider missing pid file as failure in monitor bootstrap0.98 (2019-12-02)promise: fix threshold-days in check_free_disk_space0.97 (2019-11-14)resiliencytest: Fixes for the testequeue: use BytesIO instead of StringiO as bufferpromise: Add more information on errorpromise: Support threshold-days in check_free_disk_spacepromise: Support simple threshold in check_free_disk_space0.96 (2019-09-27)promise: add check_command_execute0.95 (2019-09-09)add support for Python 3remove slapos.cloudmgrrunner: force instantiation of partitions on manual actionpromise: add check_neo_health for ERP5/NEO SR0.94 (2019-06-19)implement promise.plugin.check_file_statepromise.plugin.check_url_available: set good default timeout0.93 (2019-06-13)promise.plugin: enhance and improve check_url_availablemonitor: catch timeout rise by ssl.SSLError0.92 (2019-04-10)runner: more robust exporter script (don't check excluded files, support broken symlinks)runner: give more information when exporter script failsmonitor: escape title in OPML0.91 (2019-03-28)runner: fixed the exporter script0.90 (2019-03-20)fixed wrong upload0.89 (2019-03-20)promise.plugin: Support ATS with traffic_ctlresiliencytest: remove some slapparts hardcoded in
test code get ERP5 connection paramater from slappart0runner: API-fy inspectInstancepromise.plugin: fix test ipv6_is_faster failing when ping outside is not possible0.88 (2019-03-06)runner: add logging when we remove all instances inside a webrunner.runner: cleanup: do not use list as a variable0.87 (2019-02-28)promise check_port_listening: port parameter should be int or str, unicode is not acceptedmonitor.runpromise: exit with non zero code if a promise has failed0.86 (2019-02-06)promise: migrates old promises used in caddy-frontend SR to new format0.85 (2019-01-09)promise: add new promise to check long requestrunner: display connection parameters sorted in alphabetical order0.84 (2018-12-13)improve backupserver_check_backup promisefix resiliency testsadd UTC timezone in promises logs ("+0000")0.83 (2018-11-14)Add scripts in python to be used in the resilienceUsability improvement in the promess frameworkFix flacky tests0.82 (2018-10-26)2 new promises: check_port_listening and backupserver_check_backuprunner: add faviconresiliencytest: several fixes0.81 (2018-08-29)promise: fix bug in check_free_disk promise: use '==' instead of 'is' to compare unicode string0.80 (2018-08-29)promise: fix move check_free_disk_space promise to plugin promise0.79 (2018-08-29)promise: move check_free_disk_space promise to plugin promise with improvementsrunner: fix creating software link folder which cause rebuild of sr when /isRSReady is called0.78 (2018-08-13)promise: check_free_disk does not return error if no data is found in collect db0.77 (2018-07-31)runner: always start slaproxy service in webrunnerpromise: add new promise to check cpu load based on a threshold value0.76 (2018-04-09)promise.plugin: uses 'extra_config_dict' to send custom promise parameters0.75 (2018-03-30)monitor: reimplement monitor following the new promise design in slapgridnew promises, to check memory, check monitor bootstrap and partition statefix 'attempt to write a readonly database' in some promises and monitor.collect0.74 (2018-02-12)qemu qmpclient: add support for cpu and memory hotplugmonitor: Add a random delay of maximum 60 seconds to all monitor cron jobsmonitor: use checkpromise method from slapos.core package0.73 (2017-10-06)make monitor.collect use slapos.collect.db API'snew promise to monitor memorypromise: fix most issues in apachedex and slowquery0.72 (2017-09-25)securedelete: add securedelete script which use shred to securely wipe files0.71 (2017-09-12)promise: Add ptdigest and apachedex result check promises with tests0.70 (2017-08-18)monitor: Handle empty files case before parse json.monitor: Prevent to run collect when another instance is runningmonitor: Set timeout when getting sub monitor titlemonitor: Fix the report content outputmonitor: Allow to view and modify empty monitor config parameterqemu backup: Check running job exit after a timeoutapachedex: Initial commit of the wrapperagent: Handle not found Software Installationagent: test_mapping should containts all the tests on each groups0.69 (2017-06-01)promise.check_error_on_apache_log: Fix partial line parsers.promise.apache_mpm_watchdog: Implement watchdog for apache0.68 (2017-05-11)slapos-kill: exclude self0.67 (2017-04-18)promise.is_process_older_than_dependency_set: Verbose a bit of information when fail.monitor: Bug fix _every_xx_hourmonitor: Add missingok for log rotate, if he has no file to rotate.monitor: Automatic build statistics on promise hisotry for archive.monitor: Logrotate can call this script multiple times0.66 (2017-04-02)promise: Include promises for test networks and apache log checks.0.65 (2017-01-23)checkfeedaspromise: returns more information on failuremonitor: run command with low priority using nice0.64 (2017-01-31)monitor: merge status2rss with monitor state generation script in order to have consistent resultgeneratefeed: remove useless items instead of crashing0.63 (2017-01-09)monitor: Allow to change promise timeout from UI0.62 (2016-12-31)monitor_config_write: CORS apache conf should be rendered from a slapos templaterunner: user's git information can now contain utf-8 charactersmonitor: fix cleanup of monitor promises result folder0.61 (2016-10-26)monitor: allow to specify a second promise folderpubhubsub: new argument allowing to run the executable several times in case of failure0.60 (2016-10-05)monitor: write pid to file while processingmonitor: run all promises in a single crontask with timeout of 12 seconds per promises.0.59 (2016-09-16)runner: add testsAdd scripts to generate RSS feeds and to create promises on RSS feedsnotifier: new arguments to generate verobose logging0.58 (2016-09-07)monitor: multiple fixes and improvementsresilient: fix testsagent: fix console script0.57 (2016-08-09)monitor: Included monitor code with unit tests.0.56 (2016-07-25)slaprunner: remove create user at first launch, clone repository can be done by slaprunner at startup.Remove slapos.builder and slapos.monitor as it only contains unused and dead code.0.55 (2016-03-01)Switch to mysqlclient, that is a maintained fork of MySQL-python.Add an option in is-process-older-than-dependency-set to kill the process if older than dependency set.0.54 (2016-02-25)Added is-process-older-than-dependency-set script.0.53 (2015-11-26)networkbench: Minor fixes0.52 (2015-09-02)testagent:  Fully reimplementationnetworkbench: Accept external configurations and split file logs0.51 (2015-08-17)networkbench:  Include ping6 and TTFB support0.50 (2015-07-20)slaprunner: Allow to configure the window title through 'instance-name' parameter.0.49 (2015-07-17)slaprunner: filter out useless file extensions in treeview0.48 (2015-04-09)slaprunner: Support command exiting with non 0 status in minishell0.47.3 (2015-02-17)resiliency tests: hide output of slapos node instance.equeue: move takeover detection in thread lock.add erp5 resiliency test suite.harcode slaprunner request to use dev branch for erp5.slaprunner resiliency test suite: compatibility with python 2.7.9.resiliency test suite: do takeover from web interface.0.47.2 (2015-02-14)equeue: prevent to run import script if takeover-triggered file exists.Advertise development of new version.0.47.1 (2015-02-05)equeue: unlock lockfile at boot.0.47 (2015-02-05)runner: fix syntax when stopping all processes.equeue: realtime logging.0.46.1 (2015-01-30)runner: enhance software/instance explanation to user.runner: do not remove whole instance root directory, but only instance home directories.slaprunner resilient test: update API about slapgridResult.0.46.0 (2015-01-16)slaprunner: don't shutdown supervisord.slaprunner: Simplify process management and make sure 'process killer' handler works.webrunner: add getConnectionParameter API.equeue: create a lockfile when running importer command.Equeue: do not register command in database if it was not successful.0.45.3 (2015-01-12)slaprunner: Quick and dirty support for ace editor autocompletion0.45.2 (2014-12-02)resiliencytest framework: adapt abstract test suite to work with a single clone.resiliency test suites: send log to erp5 master.KVM resiliency test suite: use test framework0.45.1 (2014-11-24)Fix logrotate and introduce a delay fornetworkbench.0.45.0 (2014-11-24)Resiliency tests: allow to use from inside of a classical erp5testnode.0.44.0 (2014-11-22)Add newnetworkbenchcommand.0.43.0 (2014-11-13)Add newslapos-killcommand, deprecatingkillpidfromfileMakepycurlegg an optional dependency0.42.0 (2014-10-28)The egg providescheck-web-page-http-cache-hitscript to check if a web page is well cached by Squid or TrafficServer.0.41.0 (2014-10-20)The egg providesis-local-tcp-port-openedscript to check if an ip address + port are currently opened on the local machine by reading in /proc/net/tcp and /proc/net/tcp6.0.40.4 (2014-10-14)The equeue server now supports additional parameters to the callbacks, separated by \0 in the JSON protocol.The pubsubnotifier accepts a ==transaction-id parameter. If missing, the current timestamp is used.The pubsubnotifier then provides the transaction-id value in the POST call to /notify.0.40.3 (2014-10-13)Slaprunner: stop proxy when removing instances content. This fixes the "partition11 table does not exist" error.0.40.2 (2014-09-25)Slaprunner: Various bug fixSlaprunner: Update supervisorctl interaction to work with latest slapos.core revision0.40.1 (2014-09-02)Slaprunner: Fix a bad behaviour within slapos0.40 (2014-09-01)Slaprunner: Supervisord runs slapos commandsSlaprunner: fixes for git repositories management0.39.5 (Unreleased)Slaprunner: Add slave support test.0.39.4 (2014-08-11)Slaprunner: Add a minishell feature, using far less bandwith than shellinaboxSlaprunner: Slapgrid state is correctly rendered while compilingSlaprunner: consider new parameter's values without needing to restartSlaprunner: slapgrid's last result is always accessible through a stored fileSlaprunner: Fix dangerous bug concerning custom parametersSlaprunner: various fixes and improvements0.39.3 (2014-07-11)Slaprunner: Add folder to favoritesSlaprunner: Fix https git clone not possible if no loginSlaprunner: Improve editor full windowPubsub: More explicit RSS displayed in reversed chronoligical orderPubsub-notifier: add loggerSlaprunner: Various fixes0.39.2 (2014-05-16)Slaprunner: Add tab supportSlaprunner: Add upload fileSlaprunner: Improve editor full windowSlaprunner: Various fixes0.39.1 (2014-04-30)Slaprunner: Various fixes0.39 (2014-02-20)Slaprunner: new web interface designSlaprunner: one function handle both "run software" and "run instance"Slaprunner: building and deploying can be customizedSlaprunner: adds a multi-user featureSlaprunner: add fullscreen mode for text editionSlaprunner: direct access to monitoring of running instance, if it exists0.38.1 (2013-12-06)Slaprunner: do not delete proxy.db on each run software0.38 (2013-12-03)Slaprunner: adds an integrated shellSlaprunner: uses basic authentificationSlaprunner: adds automated deployment of a Software ReleaseSlaprunner: flask development server replaced by Gunicorn, a WSGI serverSlaprunner: new test scenario for auto-deploymentRunner resiliencytestsuite: adds basic auth supportRunner resiliencytestsuite: tests can be done on only one Slapos node0.37.4 (2013-10-15)Improve QEMU QMP wrapper by adding drive-backup method and other helpers.0.37.3 (2013-10-10)pubsub: don't swallow output of subprocess to allow debug.0.37.2 (2013-10-10)Add QEMU QMP wrapper.KVM resiliency test: update docstring about how to setup disk image.KVM resiliency test: change key for each clone.0.37.1 (2013-10-03)pubsub notifier: handle timeout and other connection errors.equeue: cast str(timestamp) to please gdbm.0.37 (2013-09-30)equeue: log output of subprocess.slaprunner: don't send 200 when login is bad.Improve reliability of resiliency tests.0.36 (2013-09-05)Add resiliency test system.Add kvm resiliency test suite.Add slaprunner resiliency test suite.slaprunner: don't tell user false assumptions.slaprunner: allow profile/instance to be run by GET.slaprunner: remove boring username/password length check.0.35.1 (2013-08-05)slaprunner: add support for reverse proxy.0.35.0 (2013-07-08)Resiliency: equeue is rewriten extending SocketServer.ThreadingUnixStreamServer. So far outputs of commands invoked by equeue are redirected to /dev/null to avoid locking the whole process.Add htpasswd script. This script is meant to produce htpasswd file without installing apache.slaprunner: Grammar improvements.0.34.0 (2013-03-28)runner: Log textarea is resizable.runner: Fix bug where slaproxy hangs: Redirect slapproxy output to stdout.lampconfigure: Add support for running SQL scripts in lampconfigure.runner: software_type now survives to slaprunner reboot.0.33.1 (2013-02-12)runner: Add default values for git repository while cloning.runner: Don't remove git repo if error occurred during clone.runner: Fix bug where software_type of main instance isimpossibleto set reliably.0.33 (2013-01-24)agent: run tests in order.slaprunner: Add process killer killing all processes when instance is stopped.slaprunner: slapgrid-cp will be run at startup.slaprunner: Improved design.slaprunner: Don't set arbitrary limitations on the complexity of password.0.32 (2012-12-13)pubsub: support multiple notifications and callbacks.pubsub: print/return errors from subprocess or notifications.agent: ignore SR status, retrieve instance status with .getStatus().slapmonitor: added slapmonitor-xml function + entry point.0.31.1 (2012-10-02)slaprunner: Add new file manager.slaprunner: Ask confirm before delete important folder.0.31 (2012-10-02)slaprunner: Add correct support for software_type.0.30 (2012-08-29)Add promise support on slapcontainer script.0.29.1 (2012-08-23)Minor fix on slapcontainer.0.29 (2012-08-22)Add slapcontainer stateless script.0.28.2 (2012-08-17)Slaprunner: Move password, process pid and config files to etc_dir0.28.1 (2012-08-15)Update ACE to 08.11.2012 and JQuery to 1.8.0.0.28 (2012-08-09)slaprunner: add support for authentication management.0.27 (2012-08-07)pubsub: add support for multiple notification servers.0.26.1 (2012-07-09)runner: Add support to slapos.core 0.26.20.26 (2012-07-06)slaprunner now supports authentication and instance parameters.Redisign of Test Agent so that it supports proper instantiation test and report.0.25 (2012-06-19)slapmonitor is fully functional.0.24 (2012-04-19)Slap Test Agent : test software release installation and instance allocation0.22 (2012-04-05)Install argparse if missing0.21 (2012-04-04)zodbpack : simple tool to pack ZODB0.20 (2012-03-28)slaprunner : fix recursive include of .gif files0.19 (2012-03-28)slaprunner : Add Support for more instances at the same time.slaprunner : Add slapgrid process control and management.slaprunner : Add Cloud9 IDE.slaprunner : Add develop location of cloned git repository.0.18 (2012-02-23)slaprunner : Add git commit and push support.slaprunner : Solve UI problems in project management.slaprunner : Solve bug where ACE modifies carrier returns.slaprunner : Add ajax view of build log.slaprunner : Allow to edit any file in repository.0.17 (2012-02-10)slapbuilder: Update slapbuilder to new OpenSuse 12.1.slapbuilder: Added automatic IPv6 support thanks to ViFiB OpenVPN.slapbuilder: Added virtual machine support.slapbuilder: Added "One Disk" feature.0.16 (2012-01-20)Fix MANIFEST.in0.15 (2012-01-20)slaprunner: Adding project and git repository management0.14 (2012-01-09)onetimeupload: SSL should be provided by an http frontend0.13 (2011-12-30)builder: fix usage of mkstemp0.12 (2011-12-29)builder: dynamic offset calculation0.11 (2011-12-23)equeue: Add execution queue server.pubsub: Add pubsub server using pubsubhubbub for notifications.0.10 (2011-11-07)slapbuilder: Support banging.lamp : Fix bug when using file_token option0.9 (2011-10-29)Add LAMP tool to manage lamp instances.Add onetimedownload : a simple script which download
any http resource and save it into a specified file_path.0.8 (2011-10-24)Add new style to runner.0.7 (2011-10-17)generic: Added systool with killpidfromfile.slaprunner: Avoid error when trying to remove instance directory
containing not writable directories.0.6 (2011-07-13)slapbuilder: Typo during PKI repository creation fixed.0.5 (2011-07-12)slapbuilder: Do not play too much with buildout, SlapOS installation
is responsible for correctly featured zc.buildout0.4 (1011-07-11)Update slaprunner to run on newer slapos (requires at least 0.10 to work).0.3 (2011-06-27)slapbuilder: use slapos provided networkcached buildout.slapbuilder: fix network issues on busy networks ("Neighbour table overflow.")0.2 (2011-06-10)In slapbuilder: drop to 100 partitions by default, move out PKI from USB key to real disk.0.1 (2011-05-27)Created slapos.toolbox from slapos.tool.builder, slapos.tool.runner,
slapos.tool.cloudmgr, slapos.tool.ontimeupload and slapos.tool.networkcache.