Package: lst-pressure
lst-pressurePython module for calculating LST pressure based on scheduled observationsMotivationDesignNormalizing intervalsCustomising interval calculationsUsage and installationCommand line interface (CLI)CLI ExamplesDirect API usageLocal developmentPyEnvcsvkitTestingPublishingMotivationObservation blocks encompass various time-related constraints, notably the Local Sidereal Time (LST) window of a source rise/fall, and some constraints related to UTC. As LST and UTC time can diverge over a year, it can be hard to quickly identify candidate UTC times based on LST ranges in conjunction with constraints related to sun/UTC.This library should facilitate easily comparing intervals defined in LST with intervals defined in UTC. Essentially this library is an opinionated wrapper over a Python implementation of aninterval treecalledintervaltree.DesignEach UTC day includes intervals in which an observation can be scheduled. These intervals are called:ALL_DAYSUNRISE_SUNSETSUNSET_SUNRISENIGHTTo identify the UTC intervals that satisfy scheduling constraints of observations, generate a list of LST intervals overNdays, that are labelled with UTC information. For example:[ lst_t0 : lst_t1 ]: { type: <SUNRISE_SUNSET>, utc_0, utc_1 }
[ lst_t0 : lst_t1 ]: { type, utc_0, utc_1 }
[ lst_t0 : lst_t1 ]: { type, utc_0, utc_1 }
[ lst_t0 : lst_t1 ]: { type, utc_0, utc_1 }
[ lst_t0 : lst_t1 ]: { type, utc_0, utc_1 }
... etcThen, for an observation with an LST window defined, query that list for all intervals that overlap the observation LST window. The resultant list can then be filtered by UTC-related information such as sunset/sunrise and other constraints (i.e. observation duration).Noted that the LST window in the context of this package defines possiblestarttimes of observations. It's up to the user to ensure that the start time of an observation takes into account the required observation duration.The result is a list of UTC calendar days that a particular observation can be scheduled along with UTC start/end times for applicable observations. Done for many observations, it's easy to create a UTC calendar with potential observation events that can serve as the basis for more complicated scheduling logic (for example, picking the best observation to schedule on a particular day).Normalizing intervalsIntervals should strictly havet1 > t0, which means that in some cases intervals defined by two LST times must be normalized. For example, an LST range that begins on day0 and ends on day1 will have a start time that is higher than the end time. In this case, add a full LST day to the end time at both indexing time and at query time. Likewise, when an interval - for exampleNIGHT- begins on the preceding day, the portion of the interval that falls on the current day is indexed.For any given day, intervals are created starting at 00:00, and spilling over into the next day. This means that there are overlapping intervals when considering multiple days. For example, since for day = N there are two night intervals, these intervals are actually the same period of time:date N: N.dusk -> N.dusk -> (N + 1).dawn (captured as21 -> 34)Date (N + 1): 00:00 -> (N + 1).dawn (captured as0 -> 7)Without the additional early interval, it would be impossible to match early LST observation windows (for example02:00 -> 03:30) since early morning, calculated from the night before, is> 24.Customising interval calculationsBy default this tool will calculate intervals that correspond to sunset/sunset times as available on the OPT tool, with the "NIGHT" interval equal to the "SUNSET_SUNRISE" interval (there is no concept of a NIGHT interval when calculating intervals using the default MeerKAT provider). For more general interval calculations, you can configure the tool to calculate sun statistics and intervals using the popularAstralpackage either via a CLI flag or specifying configuration within Python:PythonfromlstpressureimportLocationProviderType,LSTConfLSTConf().LOC_PROVIDER=LocationProviderType.ASTRAL# ... See below for usage examplesCLIcatopt-csv.csv|lstpressure--loc-providerASTRALobservables# See below for more examplesUsage and installationThe lst-pressure package currently supportsPython version 3.12Command line interface (CLI)The CLI wraps theLSTmodule, allowing for running thelstpressureapplication with OPT CSV download as input. Use the lstpressure CLI outside of Python environments (requires Docker, as the distribution is as a Docker image). The CLI also allows for specifying the provider for calculation sun statistics per location. By default all sun information is relative to the MeerKAT telescope. Via the CLI you can specify arbitrary lat/long locations for calculating UTC intervals.To use, ensure thatDocker Engineis installed (or Docker Desktop for non-Linux users). It's also necessary to login to the GitHub Container Registry (ghcr.io):# Navigate to https://github.com/settings/tokens and generate a 'classic' token with the "read:packages" scope selected (copy the value)dockerloginghcr.io# Username: Your GitHub username# Password: <token value># Install the lstpressure CLIdockerpullghcr.io/ska-sa/lst-pressure_dist:latest\&&docker\run\--rm\ghcr.io/ska-sa/lst-pressure_dist:latestinstall\|bash# (or uninstall the lstpressure CLI)dockerpullghcr.io/ska-sa/lst-pressure_dist:latest\&&docker\run\--rm\ghcr.io/ska-sa/lst-pressure_dist:latestuninstall\|bash# To update, pull the latest docker image (or a specific tag)dockerpullghcr.io/ska-sa/lst-pressure_dist:latest# Run the CLIlstpressure--helpCLI Examples# Print instructionslstpressure
lstpressure<cmd>-h# Get a list of today's schedulable observationscatobservations.csv|lstpressureobs# Get a list of schedulable observations over the next 3 months (including this month)catobservations.csv|lstpressureobs--end="+2M"# Get a list of schedulable NIGHT observations over the next 3 monthscatobservations.csv|lstpressureobs--end="+2M"--filternight# Get a list of schedulable NIGHT observations over the next 3 months, aggregated on a yes/no basiscatobservations.csv|lstpressureobs--end="+2m"--filternight--aggregatelst-pressure-3m# Or split the above steps for bespoke SQL aggregationcatobservations.csv|lstpressureobs--end="+2m"--filternight>night_obs_next_3m.csv
catnight_obs_next_3m.csv|lstpressureagg--querylst-pressure-3m# Use the DuckDB query engine by specifying a query that includes "duckdb", or including "duckdb" in a comment in a querycatnight_obs_next_3m.csv|lstpressureagg--queryobservables-by-month.duckdb# or..catnight_obs_next_3m.csv|lstpressureagg--query"select * from stdin -- duckdb"# Write bespoke aggregations (SQLite syntax, in the case of multiple statements, the last is returned)catnight_obs_next_3m.csv|lstpressureagg--query"select * from stdin;"# Interrogate what the "lst-pressure-3m" query looks likelstpressureagg--querylst-pressure-3m--echo# .. Write your own SQL in a file# Using the CLI via the Docker image requires filepaths to be specified as absolute paths, i.e. --query $(pwd)/my-report.sqlecho"select * from stdin;">my-report.sql
catnight_obs_next_3m.csv|lstpressureagg--query$(pwd)/my-report.sql# or ...catobservations.csv|lstpressureobs--end="+2m"--filternight--aggregate$(pwd)/my-report.sqlThe aggregation tool just wraps thecsvkit (csvsql)program. Other tools can easily be integrated into this CLI in the future. Two SQL engines are currently supported -DuckDBandSQLite.DuckDBis more versatile as an aggregation framework for it's support ofdynamic pivoting.Direct API usageInstall fromPyPi:pipinstalllst-pressureThen use thelstmodule (API Docsare available)importlstpressureaslstimportpandasaspd# Specify the provider for calculating sun stats (defaults to bespoke logic for MeerKAT, to match the OPT tool times)lst.LSTConf().LOC_PROVIDER=lst.LocationProviderType.MEERKATSTART="20231108"END="20231208"# Filter for specific rows in the CSVdefobservation_filter(observation:lst.Observation):iflst.LSTIntervalType.NIGHTinobservation.utc_constraints:returnTruereturnFalse# Specifying a filter on observations requires# processing the entire CSV file (and doing sun calculations# on each row). If it's slow, a lot less work is done if# you filter on inputdefinput_filter(row:pd.Series):night_only:str=row.iloc[15]ifnight_only.strip().lower()=="yes":returnTruereturnFalselst.LST(input="./observations.csv",calendar_start=START,calendar_end=END,# input_filter=input_filter,observation_filter=observation_filter,).to_csv_file(f"lst-pressure-{START}-{END}.csv")# ... Or provide input as a pandas.DataFrame instance# ... Or provide input as a list[list[Any]]It's also possible to work with the underlyingObservation,LSTCalendar,LSTIntervaltypes directly. For the most part this should not be required, however it does allow for using thelstpressure.is_observablefunction. In terms of implementation, we don't use the OPT observations download directly but instead use instances of thelstpressure.Observationclass so as to decouple the application logic from the CSV structure.importlstpressure# Define an observationobservation=lstpressure.Observation(id="some-id",lst_window_start=2,lst_window_end=12,utc_constraints=[lstpressure.lstindex.LSTIntervalType.NIGHT],duration=2)# Test if it can be scheduled on a dateis_valid=lstpressure.is_observable(observation,"20231031")# ... or within a date rangeis_valid=lstpressure.is_observable(observation,"20231031","20231105")# ... or test multiple observations against a calendarcalendar=lstpressure.LSTCalendar('20231001','20231001')is_obs1_valid=lstpressure.is_observable(Observation(...),calendar)is_obs2_valid=lstpressure.is_observable(Observation(...),calendar)is_obs3_valid=lstpressure.is_observable(Observation(...),calendar)# ... or get a list of dates that an observation can be scheduled onobservables=[o# use o.to_tuple() for CSV values, or access the interval/observation via the observable instanceforoinlstpressure.LSTCalendar("20231001","20231010",observations=[Observation(...),...]).observables()]# ... or specify the ASTRAL provider for calculationslstpressure.LSTConf().LOC_PROVIDER=lstpressure.LocationProviderType.ASTRALLocal developmentEnsure that your Python context is3.12.x, and runsource env.sh. this will install project dependencies and add./bin/lstto your path.PyEnvSo long as your current Python version is 3.12, things should 'just work'. One tool for quickly switching between various Python versions ispyenv- to install:# PyEnv builds Python versions from source, therefore build dependencies are requiredbrewinstallopensslreadlinesqlite3xzzlib# Install pyenv using the pyenv-installer (don't forget to add 'pyenv' to the load path!)curlhttps://pyenv.run|bash# Install python v3.12 using pyenvpyenvinstall-v3.12# Not macarch-x86_64pyenvinstall3.12# Mac# Create a pyenv virtual environmentpyenvvirtualenv3.12py3.12# Activate your pyenv environmentpyenvactivatepy3.12... And now your current Python context should be3.12.xcsvkitThe current output of lstpressure is quite raw, and some aggregating needs to be done on it. csvsql is a command-line tool provided by the csvkit Python library for performing SQL operations on CSV files. To install csvkit and, consequently, csvsql on your Mac, you can follow these steps:# Install csvkit using pippipinstallcsvkit# Verify the installationcsvsql--versionThis should display the version number of csvsql if the installation was successful.Now you should be able to use csvsql on your Mac.TestingTo test the codebase, runpytestin the terminal. For live testing, use thechomptask runner. Install either via Cargo (Rust), or via NPM (Node.js)sourceenv.sh
chomp--watchPublishingThe publish workflow is described in.github/workflows/publish.yml, and is triggered on tags pushed to themainbranch. The published package is available onPyPI. Make sure to first increment the version inversionbefore pushing to the release branch (please make sure that the tag version matches the version string insetup.py).