Package: tessdb-server
tessdb-server(overview)Linux service to collect measurements pubished by TESS Sky Quality Meter via MQTT. TESS stands forCristobal Garcia's Telescope Encoder and Sky Sensortessdbis being used as part of theSTARS4ALL Project.Descriptiontessdbis a software package that collects measurements from one or several
TESS instruments into database (currently a SQLite Database).It is aPython Twisted Applicationthat uses acustom Twisted library implementing the MQTT protocolDesktop applicatons may query the database to generate reports and graphs
using the accumulated, historic data. There are some reports scripts already included in the package, specially an IDA-format monthly report script.Note: The Windows version has been dropped, as it was never used.These data sources are available:individual samples (real time, configurable, 1 min. aprox between samples).Instrument should send their readings at twice the time resolution specified in the config file (in seconds).Warning: Time referencie is always UTC, not local time.INSTALLATIONRequirementsThe following components are needed and should be installed first:python 2.7.x (tested on Ubuntu Python 2.7.6) or python 3.6+Note:It is foreseen a Python 3 migration in the future, retaining Python 2.7.x compatibility.InstallationInstallation is done from GitHub:git clone https://github.com/astrorafael/tessdb.git
cd tessdb
sudo python setup.py installNote:Installation from PyPi is now obsolete. Do not use the package uploaded in PyPi.All executables are copied to/usr/local/binThe database is located at/var/dbase/tess.dbby defaultThe log file is located at/var/log/tessdb.logThe following required PIP packages will be automatically installed:twisted,twisted-mqttpyephemStart up and VerificationTypesudo tessdb -kto start the service in foreground with console output.
Verify that it starts without errors or exceptions. When done, abort it with^CFrom tessdb release 1.2.0, the background execution is handled as asystemdservice instead of the old system V style init script:`sudo systemctl start tessdb`although the oldsudo service tessdb startcommand still works.It is strongly recommended to enable the service at boot time by issuing:sudo systemctl enable tessdbCONFIGURATIONThere is a small configuration file for this service:/etc/tessdb/config(Linux)This file is self explanatory.
In special, the database file name and location is specified in this file.
Some of the properities marked in this file are marked asreloadable property. This means that this value can be changed and the process reloads its new value on the fly.Sunrise / Sunset filteringA configurable window (7 samples by default) is stored in the server for each photometer.
A filter process analyzes the middle sample in this window, if it finds all saturated values (magnitude=0)
around this middle sample (past and future), this sample is discarded, otherwise it is elegible to be written to the database.Positive: This filtering does not need to know the position where the photometer is placed.Negative: There is a time lag (Window size/2) between receiving the sample and storing it.
Furthermore, if the server crashes or is abruptly stopped, the so-called "future samples" are lost.LoggingLog file is usually placed under/var/log/tessdb.login Linux or underC:\tessdb\logfolder on Windows.
Default log level isinfo. It generates very litte logging at this level.
File is rotated by logrotateonly under Linux.OPERATIONServer Status/Start/Stop/RestartService status:sudo systemctl status tessdborsudo service tessdb statusStart Service:sudo systemctl start tessdborsudo service tessdb startStrongly recommended:Stop Service:sudo /usr/local/bin/tessdb_stopRestart Service:sudo /usr/local/bin/tessdb_restartNo don use:Stop Service:sudo systemctl stop tessdborsudo service tessdb stopRestart Service:sudo systemctl restart tessdborsudo service tessdb stopService Pause/ResumeThe server can be put inpause mode, in which will be still receiving incoming MQTT messages but will be internally enqueued and not written to the database. Also, all connections to the database are closed. This allows to perform high risk operations on the database without loss of incoming data. Examples:Compact the database using the SQLiteVACUUMpragmaMigrating data from tables.etc.Service pause/resume use internally signalsSIGUSR1andSIGUSR2.To pause the server, type:sudo tessdb_pauseand watch the log file output wittail -f /var/log/tessdb.log:2018-11-23T13:08:25+0100 [dbase#info] TESS database writer paused
2018-11-23T13:08:25+0100 [dbase#info] Closed a DB Connection to /var/dbase/tess.dbNote:The oldsudo service tessdb pausecommand do not work anymore.To resume normal operation typesudo tessdb_resumeand watch the same log file:2018-11-23T13:10:37+0100 [dbase#info] TESS database writer resumed
2018-11-23T13:10:37+0100 [dbase#info] Opened a DB Connection to /var/dbase/tess.dbService reloadDuring a reload the service is not stopped and re-reads the new values form the configuration file and apply the changes. In general, all aspects not related to maintaining the current connection to the MQTT broker or changing the database schema can be reloaded. The full list of reloadadble properties is described inside the configuration file.Typesudo systemctl reload tessdborsudo service tessdb reload.MainteinanceDatabase mainteinance is made through thetesscommand line utility, installed by the tessdb-reports package.
Mainteninance operations include:create new locationscreate new TESS instruments (manually)assign locations to instrumentsenable recording of data received from an instrumentlisting current instrumentslisting instruments not assigned to any known locationetc.DATA MODELDimensional ModellingThe data model follows the [dimensional modelling]
(https://en.wikipedia.org/wiki/Dimensional_modeling) approach by Ralph Kimball. More references can also be found inStar Schemas.The data modelThe figure below shows the layout oftessdb.Dimension TablesThey are:date_t: preloaded from 2016 to 2026time_t: preloaded, with seconds resolution (configurable)tess_t: registered TESS instruments collecting datalocation_t: locations where instruments are deployedtess_units_t: an assorted collection of unit labels for reports, preloaded with current units.tess_v: View with TESS instrument and current location. It is recommended that reporting applications use this view, instead of the underlyingtess_tandlocation_ttables.Date DimensionPretty much standard date table from dimensional modelling. Contains most used attributes plusjulian_dayspecific to Astronomy domain.Time of the day DimensionPretty much standard time of the day table from dimensional modelling. Contains well known attributes.Instrument DimensionThis dimension holds the current list of TESS instruments.The real key is an artificial keytess_idlinked to the Fact table.Themac_addresscould be the natural key if it weren't for the zero point and filter history tracking.Thelocation_idis a reference to the current location assigned to the instrument. Location id -1 denotes the "Unknown" location.modelrefers to the current TESS model.firmwarecontains the current firmware version.fovcontains the Field of View, in degrees.cover_offsetis an additional offset in mag/arcserc^2 to account for an additional optical window attenuation attached tothe TESS itself. Defaults to 0.0.channelis the current channel identifier. Default value is 0. Currently, the TESS photometer has only one channel.authorisedto allow the TESS instrument to store readings on the database. Authorization is a manual process done by thetess utility.registeredshos if the TESS instrument registered itself on the database ("Automatic") or it was done by a manual process ("Manual") using thetess utility. The default value is "Unknown" for the TESS instrument registered before adding this feature to the software. It is expected to identify these cases one by one and set them to 'Manual' or 'Automatic'.Version-controlled AttributesThese attubutes are version-controlled and a historic of these is maintained. A new change in any of them will generate a new row in thetess_ttableThezero_pointholds the current value of the instrument calibration constant. Defaults to '20.5' (uncalibrated photometer).Thefilterholds the current TESS filter (i.e. 'UVIR' or Dichroic Glass). Defaults to 'UVIR'Theazimuthandaltitudeattributes hold the photometer current orientation, in degrees. Default azimuth is 0.0 and default altitude is 90.0Version Control AttributesThese columns manage the version control of a given TESS attributes.Columnsvalid_sinceandvalid_untilhold the timestamps where the changes to version controlled attributes are valid.Columnvalid_stateis an indicator. Its values are eitherCurrentorExpired.
The current valid TESS instrument has itsvalid_stateset toCurrentand the expiration date in a far away future (Y2999).Unit dimensionThetess_units_ttable collects various flags for the fact table.(OBSOLETE) Columnsvalid_since,valid_untilandvalid_statekeep track of any units change in a similar technique as above should the units change.Location dimensionThis dimension table holds all known locations where TESS photometers are to be deployed.site. Unique site name describing the this location.contact_person. Person to account for observations.organization. Organization where the contact person belongs to or running the facilities in the location.contact_email. Contact person email address.longitudeLocation longitude in degrees. West is negative.latitude. Location latitude in degreeselevation. Location elevation in meterszipcode. Location ZIP codeprovince. Location countrycountry. Location countrytimezone. Time zone (to calculate local time) in standard format described by Wikipedia[https://en.wikipedia.org/wiki/List_of_tz_database_time_zones]sunrise&sunset. Computed attributes (oce per day) used to filter out readings in daylight.Fact TablesThey are:tess_readings_t: Accumulating snapshot fact table containing measurements from several TESS instruments.TESS devices with accelerometer will sendazimuthandaltitudevalues, otherwise they areNULL.TESS devices with a GPS will sendlongitude,latitudeandheightvalues, otherwise they areNULL.Other TablesIt is possible now to replace a given TESS-W and keep the name. This is sueful for sites that wish to announce a simbolic name for its TESS-W photometer and never change even if the device is replaced (i.e. by being broken).So now, a given namedoes notidentify a TESS-W photometer, only the MAC address does. An association table is neededname_to_mac_t: Association table to label a given TESS-W device (identified by its MAC address) to a symbolic name.Sample SQL QueriesThe following are samples queries illustraing how to use the data model. They are actually being used by the STARS4ALL projectGet a daily report of readings per instrument:#!/bin/bashsqlite3/var/dbase/tess.db<<EOF.mode column.headers onSELECT d.sql_date, i.mac_address, count(*) AS readingsFROM tess_readings_t AS rJOIN tess_t AS i USING (tess_id)JOIN date_t AS d USING (date_id)GROUP BY r.date_id, r.tess_idORDER BY d.sql_date DESC;EOFExtract a CSV (semicolon separated) with all readings for an instrument passed as a command line argument:#!/bin/bashinstrument_name=$1sqlite3-csv-header/var/dbase/tess.db<<EOFSELECT (d.julian_day + t.day_fraction) AS julian_day, (d.sql_date || 'T' || t.time) AS timestamp, r.sequence_number, l.site, i.mac_address, r.frequency, r.magnitude, i.zero_point, r.sky_temperature, r.ambient_temperatureFROM tess_readings_t AS rJOIN tess_t          AS i USING (tess_id)JOIN location_t      AS l USING (location_id)JOIN date_t          AS d USING (date_id)JOIN time_t          AS t USING (time_id)WHERE i.mac_address  IN (SELECT mac_address FROM name_to_mac_t WHERE name = "${instrument_name}")ORDER BY r.date_id ASC, r.time_id ASC;EOFShow current TESS instruments. Note that we are using thetess_vView,so that the current location info is already included.#!/bin/bashsqlite3/var/dbase/tess.db<<EOF.mode column.headers onSELECT v.name AS Name, v.mac_address AS MAC, (v.latitude || ' ' || v.longitude) AS Coordinates , (v.site || ', ' || v.location || ', ' || v.province) AS Location, v.contact_email as User, v.zero_point as ZP, v.filter as FilterFROM tess_v AS vWHERE v.valid_state = "Current"ORDER BY v.name ASC;EOFShow TESS instruments changes (zero point and/or filter)#!/bin/bashsqlite3/var/dbase/tess.db<<EOF.mode column.headers on;SELECT i.name AS Name, i.zero_point as ZP, i.filter as Filter, i.valid_since AS Since, i.valid_until AS Until, i.valid_state AS 'Change State'FROM tess_t AS iORDER BY i.name ASC, i.valid_since ASC;EOFShow the time span of readings per TESS#!/bin/bashsqlite3/var/dbase/tess.db<<EOF.mode column.headers on;SELECT i.mac_address, MIN(d.sql_date || 'T' || t.time || 'Z') AS earliest, MAX(d.sql_date || 'T' || t.time || 'Z') AS latestFROM tess_readings_t AS rJOIN tess_t          AS i USING (tess_id)JOIN location_t      AS l USING (location_id)JOIN date_t          AS d USING (date_id)JOIN time_t          AS t USING (time_id)GROUP BY i.mac_address;EOFShow locations not assigned to any TESS#!/bin/bashsqlite3/var/dbase/tess.db<<EOF.mode column.headers on;SELECT l.siteFROM location_t        AS lLEFT OUTER JOIN tess_t AS i USING (location_id)WHERE i.mac_address IS NULL;EOFMQTT PAYLOAD INFORMATIONPayloads are transmitted in JSON format, with the format described below.Published on  topic 'STARS4ALL/{channel}/reading'Field nameTypeUnitsOptionalDescriptionseqint-mandSequence number. If possible use 32 bits. The sequence number will start in 1 at each device reboot.namestring-mandInstrument friendly name. Should be unique as it identifies the device.freqfloatHzmandRaw reading as a frequency with 3 decimal digits precision (millihertz) NNNNN.NNNmagfloatmag/arcsec^2mandat.Visual magnitude (formulae?) corresponding to the raw reading). Transmitted up to two decimal places NN.NNtambfloatºCmandat.Ambient Temperature. Transmitted up to one decimal place.tskyfloatºCmandat.Sky Temperature. Transmitted up to one decimal place.wdBmintdBmoptWiFi Received Signal Strength.azintdegoptPhotometer optical axis Azimuth sent only on instruments with accelerometer.altintdegoptPhotometer optical axis Altitude (angle): sent only on instruments with accelerometer.latfloatdegoptInstrument latitude. Only sent by instruments with GPS integration.longfloatdegoptInstrument longitude. Only sent by instruments with GPS integration.heightfloatmetersoptInstrument height above the sea level. Only sent by instruments with GPS integration.revint-mandPayload data format revision number. Current version is 1.tstampstringUTCoptTimestamp,“YYYY-MM-DDTHH:MM:SS” format. Must be UTC.Published on  topic 'STARS4ALL/register'Field nameTypeUnitsOptionalDescriptionnamestring-mandInstrument friendly name. Should be unique as it identifies the device.macstring-mand.Device MAC address, format “xx:yy:zz:rr:ss:tt”calibfloatmag/arcsec^2mandPer-device Zero Point. Transmitted as NN.NN floating point.revint-mandPayload data format revision number. Current version is 1.chanstring-optChannel where this instrument will publish its readings.OPERATION & MAINTENANCEPause & resumeSince the current database used is SQLite - a single user database - you need to pause tessdb-server if dealing directly with the database
like issuing SQL commands directly or using thetesscommand line utilityUse/usr/local/bin/tessdb_pauseand/usr/local/bin/tessdb_resumeto coordinate your direct interactions to the database with tessdb-server.ReloadSince tessdb-server maintains a RAM cache of photometers data, sometesscommand requires a server reload to rfefress the cache.
Thetessutility wanrs you when this is necessary.RestartThe newest filter operation mode in tessdb-server maintains a sliding window of photometers samples before writting to database
If it is necessary to restart the server, use/usr/local/bin/tessdb_restartinstead ofservice tessdb restart. This will ensure that the lastest
readings are stored in the database.Thetessutilitytessis a command line utility to perform some common operations on the database without having to write SQL statements. As this utility modifies the database, it is necessary to invoke it within usingsudo. Also, you should ensure that the database is not being written bytessdbto avoiddatabase is lockedexceptions, either by using it at daytime or by pausing thetessdbservice with/usr/local/bin/tessdb_pauseand then resume it with/usr/local/bin/tessdb_resume.