Package: healpix-alchemy
To acknowledge HEALPix Alchemy in a scholarly article, please cite the following two references.HEALPix Alchemy paperSinger, L. P., Parazin, B., Coughlin, M. W., et al. (2022). "HEALPix Alchemy: Fast All-Sky Geometry and Image Arithmetic in a Relational Database for Multimessenger Astronomy Brokers."Astronomical Journal163209.https://doi.org/10.3847/1538-3881/ac5ab8Code archive on ZenodoSinger, L. P., Parazin, B., Coughlin, M. W., et al. (2022). "skyportal/healpix-alchemy: Version 1.0.1."https://doi.org/10.5281/zenodo.5768564HEALPix AlchemyThehealpix_alchemyPython package is an extension for theSQLAlchemyobject relational mapper. It adds region and image arithmetic toPostgreSQL(version 14 and newer) databases. It accelerates queries between point clouds,
regions, and images (sometimes known in the geospatial community as rasters) by
storingmulti-orderHEALPixindices in PostgreSQL'srange types.Thehealpix_alchemyproject is designed for astronomy applications,
particularly for cross-matching galaxy catalogs, observation footprints, and
all-sky images likegravitational-wave probability sky mapsor even dust
maps. However, it could be used in any context in which geometry is embedded on
a unit sphere.healpix_alchemyis lean and minimalist because it leverages several existing
projects: it consists of little more than a few lines of glue code to bind
togetherMOCPy,SQLAlchemy, and PostgreSQL'srange types.healpix_alchemyserves a purpose similar to full-featured astronomy-focused
spatial extensions likeQ3C,H3C, andpg_healpix, and geospatial
extensions likePgSphereandPostGIS. What setshealpix_alchemyapart
from these is that it is written in pure Python and requires no server-side
database extensions. Consequently,healpix_alchemycan be used with managed
PostgreSQL databases in the cloud likeAmazon RDSandGoogle Cloud SQL.Theory of operationHEALPix basicsHEALPixis a scheme for subdividing and indexing the unit sphere, originally
described byGórski et al. (2005). Although it was originally designed for
cosmic microwave background analysis, it has found many uses in astronomy,
particularly throughmulti-order coverage (MOC)maps andhierarchical progressive surveys (HiPS)used extensively in theAladinastronomical information system. It is also used by LIGO and Virgo to store and
communicategravitational-wave probability sky maps.HEALPix can be thought of as a tree. At the lowest resolution, level 0, HEALPix
subdivides the sphere into 12 equal-area base tiles, assigned integer indices
0 through 11. At level 1, each of the 12 base tiles is subdivided into 4 tiles.
Every subsequent level divides each of the preceding level's tiles into 4 new
tiles. At a given level, each of the base pixels has been divided into
4levelpixels (nside= 2levelpixels on each side).
Thus there arenpix= 12×4levelpixels at a given resolution,
assigned integer indices from 0 through (npix-1). This is called the NESTED
indexing scheme. (There is also a RING indexing scheme in which the indices
advance from east to west and then from north to south.)A HEALPix tile, a node in the HEALPix tree, is fully addressed by three pieces
of information: the indexing scheme (RING or NESTED), the resolution level
(levelor equivalentlynside), and the pixel index (ipix, an integer
between 0 andnpix-1).The image below, reproduced fromhttps://healpix.jpl.nasa.gov, illustrates the
first 4 levels of refinement of a HEALPix grid.HEALPix interval setsA region on the sphere can be encoded by a collection of disjoint HEALPix
tiles, potentially at a mix of different resolution levels. Typically, large
low-resolution tiles are used on the interior of the region, and small
high-resolution tiles are used on the boundary. This is called amulti-order coverage (MOC)map. An example, reproduced from theMOCPy documentation, is shown below.Much like MOCs, LIGO/Virgo/KAGRA gravitational-wave probability sky maps are
stored as multi-resolution HEALPix data sets, but with a vector of
floating-point values attached to each tile. A multi-order refinement mesh from
an example sky map is shown below, reproduced fromSinger & Price (2015).Reinecke & Hivon (2015)introduced HEALPix interval sets as an
alternative encoding of MOCs that enables fast and simple unions,
intersections, and queries. In an interval set, each HEALPix tile is described
by the interval of pixel indices at some very high resolutionlevelmaxthat are descendents of that tile. In an interval set, a
region is encoded as a disjoint collection of such intervals. A tile with a
NESTED address given by (level,ipix) may be described as the half-open
interval[ipix4levelmax-level,
(ipix+ 1) 4levelmax-level).We uselevelmax= 29 because this is the highest resolution at
which pixel indices can be stored in a signed 64-bit integer. At this
resolution, each pixel is scarcely 0.4 milliarcseconds across.The interval set representation is adventageous because there are simple and
fast algorithms for interval arithmetic and set operations. Interval analysis
appears in a suprising variety of scientific contexts fromgenomicstogravitational wave data quality. Because of the many
business applications of interval arithmetic, intervals are also supported in
thePostgreSQLdatabase through itsrange types.Spatial primitives inhealpix_alchemyThehealpix_alchemypackage provides twocustom column typesforSQLAlchemy:healpix_alchemy.PointThis class represents a point. A column of this type could store the positions
of galaxies in a catalog. Under the hood, it is just aBIGINT.Wherever you need to bind a Python value to ahealpix_alchemy.Point, you may
provide any one of the following:an instance ofastropy.coordinates.SkyCoorda sequence of twoastropy.units.Quantityinstances with angle units, which
will be interpreted as the right ascension and declination of the point in
theICRSframean integer representing the HEALPix NESTED index of the point atlevel=levelmaxhealpix_alchemy.TileThis class represents a HEALPix tile. A table containing a column of this type
and a foreign key could store MOCs or gravitational-wave probability maps.
Under the hood, it is just anINT8RANGE.Wherever you need to bind a Python value to ahealpix_alchemy.Tile, you may
provide any one of the following:A single integer which will be interpreted as the address of the tile in theUNIQHEALPix indexing schemeA sequence of two integers, which will be interpreted as the lower and upper
bounds of the right-half-open pixel index interval atlevel=levelmaxA string like'[1234,5678)'InstallationYou can installhealpix_alchemyfrom the Python Package Index usingpip:$pipinstallhealpix-alchemyDevelopment InstallationContributions are welcome! This package uses thePoetrypackaging and
dependency tool andpytestfor unit tests. To installhealpix_alchemyin a
development environment, follow these instructions.InstallPoetryby following theofficial Poetry installation instructions.Clone this repository:$gitclonehttps://github.com/skyportal/healpix-alchemy.git$cdhealpix-alchemyInitialize the Poetry-managed virtual environment withhealpix_alchemyand all of its dependencies installed by running this command:$poetryinstallNow, you can enter a shell inside the virtual environment by running:$poetryshellTo run the test suite, including the examples in this README file, run this
command inside the Poetry shell:$pytestExampleFirst, some imports:>>>fromsqlalchemyimportorm>>>importsqlalchemyassa>>>importhealpix_alchemyashaSet up tablesThis example will use the SQLAlchemy declarative extension for describing table
schema using Python classes.SQLAlchemy needs to know the name for each table. You can provide the name by
setting the__tablename__attribute in each model class, or you can
create a base class that generates the table name automatically from the class
name.>>>@orm.as_declarative()...classBase:......@orm.declared_attr...def__tablename__(cls):...returncls.__name__.lower()Each row of theGalaxytable represents a point in a catalog:>>>classGalaxy(Base):...id=sa.Column(sa.Text,primary_key=True)...hpx=sa.Column(ha.Point,index=True,nullable=False)Each row of theFieldtable represents a ZTF field:>>>classField(Base):...id=sa.Column(sa.Integer,primary_key=True)...tiles=orm.relationship(lambda:FieldTile)Each row of theFieldTiletable represents a multi-resolution HEALPix tile
that is contained within the corresponding field. There is a one-to-many
mapping betweenFieldandFieldTile.>>>classFieldTile(Base):...id=sa.Column(sa.ForeignKey(Field.id),primary_key=True)...hpx=sa.Column(ha.Tile,primary_key=True,index=True)Each row of theSkymaptable represents a LIGO/Virgo HEALPix
localization map.>>>classSkymap(Base):...id=sa.Column(sa.Integer,primary_key=True)...tiles=orm.relationship(lambda:SkymapTile)Each row of theSkymapTiletable represents a multi-resolution HEALPix
tile within a LIGO/Virgo localization map. There is a one-to-many mapping
betweenSkymapandSkymapTile.>>>classSkymapTile(Base):...id=sa.Column(sa.ForeignKey(Skymap.id),primary_key=True)...hpx=sa.Column(ha.Tile,primary_key=True,index=True)...probdensity=sa.Column(sa.Float,nullable=False)Finally, connect to the database, create all the tables, and start a session.>>>engine=sa.create_engine('postgresql://user:password@host/database')>>>Base.metadata.create_all(engine)>>>session=orm.Session(engine)Populate with sample dataLoad the2MASS Redshift Surveyinto theGalaxytable. This catalog contains
44599 galaxies.It may take up to a minute for this to finish. Advanced users may speed this up
significantly by vectorizing the conversion fromSkyCoordto HEALPix indices
and usingSQLAlchemy bulk insertion.>>>fromastropy.coordinatesimportSkyCoord>>>fromastroquery.vizierimportVizier>>>vizier=Vizier(columns=['SimbadName','RAJ2000','DEJ2000'],row_limit=-1)>>>data,=vizier.get_catalogs('J/ApJS/199/26/table3')>>>data['coord']=SkyCoord(data['RAJ2000'],data['DEJ2000'])>>>forrowindata:...session.add(Galaxy(id=row['SimbadName'],hpx=row['coord']))>>>session.commit()Load the footprints of theZwicky Transient Facilityfields into theFieldandFieldTiletables.It may take up to a minute for this to finish. Advanced users may speed this up
significantly by usingSQLAlchemy bulk insertion.>>>fromastropy.tableimportTable>>>fromastropy.coordinatesimportSkyCoord>>>fromastropyimportunitsasu>>>url='https://raw.githubusercontent.com/ZwickyTransientFacility/ztf_information/9fd0ba8842709f42a134c88827309ccab728fcb7/field_grid/ztf_field_corners.csv'>>>forrowinTable.read(url):...field_id=int(row['field'])...corners=SkyCoord(row['ra1','ra2','ra3','ra4'],...row['dec1','dec2','dec3','dec4'],...unit=u.deg)...tiles=[FieldTile(hpx=hpx)forhpxinha.Tile.tiles_from(corners)]...session.add(Field(id=field_id,tiles=tiles))>>>session.commit()Load a sky map for LIGO/Virgo eventGW200115_042309(S200115j) into theSkymapandSkymapTiletables.>>>url='https://gracedb.ligo.org/apiweb/superevents/S200115j/files/bayestar.multiorder.fits'>>>data=Table.read(url)>>>tiles=[SkymapTile(hpx=row['UNIQ'],probdensity=row['PROBDENSITY'])forrowindata]>>>session.add(Skymap(id=1,tiles=tiles))>>>session.commit()Last, runANALYZEto prepare the data for use:>>>session.execute(sa.text('ANALYZE'))<sqlalchemy.engine.cursor.CursorResult object at 0x...>Sample QueriesWhat is the area of each field?>>>query=sa.select(...FieldTile.id,sa.func.sum(FieldTile.hpx.area)...).group_by(...FieldTile.id...).limit(...5...)>>>forid,areainsession.execute(query):...print(f'Field{id}has area{area:.3g}sr')Field 199 has area 0.0174 srField 200 has area 0.0174 srField 201 has area 0.0174 srField 202 has area 0.0174 srField 203 has area 0.0174 srHow many galaxies are in each field?>>>count=sa.func.count(Galaxy.id)>>>query=sa.select(...FieldTile.id,count...).filter(...FieldTile.hpx.contains(Galaxy.hpx)...).group_by(...FieldTile.id...).order_by(...count.desc()...).limit(...5...)>>>forid,ninsession.execute(query):...print(f'Field{id}contains{n}galaxies')Field 1739 contains 343 galaxiesField 699 contains 336 galaxiesField 700 contains 311 galaxiesField 225 contains 303 galaxiesField 1740 contains 289 galaxiesWhat is the probability density at the position of each galaxy?>>>query=sa.select(...Galaxy.id,SkymapTile.probdensity...).filter(...SkymapTile.id==1,...SkymapTile.hpx.contains(Galaxy.hpx)...).order_by(...SkymapTile.probdensity.desc()...).limit(...5...)>>>forid,pinsession.execute(query):...print(f'{id}has prob. density{p:.5g}/sr')2MASX J02532153+0632222 has prob. density 20.701/sr2MASX J02530482+0555431 has prob. density 20.695/sr2MASX J02533119+0628252 has prob. density 20.669/sr2MASX J02524584+0639206 has prob. density 20.656/sr2MASX J02534120+0615562 has prob. density 20.567/srWhat is the probability contained within each field?>>>area=(FieldTile.hpx*SkymapTile.hpx).area>>>prob=sa.func.sum(SkymapTile.probdensity*area)>>>query=sa.select(...FieldTile.id,prob...).filter(...SkymapTile.id==1,...FieldTile.hpx.overlaps(SkymapTile.hpx)...).group_by(...FieldTile.id...).order_by(...prob.desc()...).limit(...5...)>>>forid,probinsession.execute(query):...print(f'Field{id}probability is{prob:.3g}')Field 1499 probability is 0.165Field 1446 probability is 0.156Field 452 probability is 0.154Field 505 probability is 0.0991Field 401 probability is 0.0962What is the combined area of fields 1000 through 2000?In the next two examples, we introducehealpix_alchemy.func.union()which
finds the union of a set of tiles. Because it is an aggregate function, it
should generally be used in a subquery.>>>union=sa.select(...ha.func.union(FieldTile.hpx).label('hpx')...).filter(...FieldTile.id.between(1000,2000)...).subquery()>>>query=sa.select(...sa.func.sum(union.columns.hpx.area)...)>>>result=session.execute(query).scalar_one()>>>print(f'{result:.3g}sr')9.33 srWhat is the integrated probability contained within fields 1000 through 2000?>>>union=sa.select(...ha.func.union(FieldTile.hpx).label('hpx')...).filter(...FieldTile.id.between(1000,2000)...).subquery()>>>prob=sa.func.sum(SkymapTile.probdensity*(union.columns.hpx*SkymapTile.hpx).area)>>>query=sa.select(...prob...).filter(...SkymapTile.id==1,...union.columns.hpx.overlaps(SkymapTile.hpx)...)>>>result=session.execute(query).scalar_one()>>>print(f'{result:.3g}')0.837What is the area of the 90% credible region?>>>cum_area=sa.func.sum(...SkymapTile.hpx.area...).over(...order_by=SkymapTile.probdensity.desc()...).label(...'cum_area'...)>>>cum_prob=sa.func.sum(...SkymapTile.probdensity*SkymapTile.hpx.area...).over(...order_by=SkymapTile.probdensity.desc()...).label(...'cum_prob'...)>>>subquery=sa.select(...cum_area,...cum_prob...).filter(...SkymapTile.id==1...).subquery()>>>query=sa.select(...sa.func.max(subquery.columns.cum_area)...).filter(...subquery.columns.cum_prob<=0.9...)>>>result=session.execute(query).scalar_one()>>>print(f'{result:.3g}sr')0.277 srWhich galaxies are within the 90% credible region?>>>cum_prob=sa.func.sum(...SkymapTile.probdensity*SkymapTile.hpx.area...).over(...order_by=SkymapTile.probdensity.desc()...).label(...'cum_prob'...)>>>subquery=sa.select(...SkymapTile.probdensity,...cum_prob...).filter(...SkymapTile.id==1...).subquery()>>>min_probdensity=sa.select(...sa.func.min(subquery.columns.probdensity)...).filter(...subquery.columns.cum_prob<=0.9...).scalar_subquery()>>>query=sa.select(...Galaxy.id...).filter(...SkymapTile.id==1,...SkymapTile.hpx.contains(Galaxy.hpx),...SkymapTile.probdensity>=min_probdensity...).limit(...5...)>>>forgalaxy_id,insession.execute(query):...print(galaxy_id)2MASX J02424077-00004782MASX J02352772-09212162MASX J02273746-01092262MASX J02414523+00263542MASX J20095408-4822462