JVO Skynode Implementation Experience

Yuji SHIRASAKI
National Astronomical Observatory ofJapan, JVO

VOQL session

Contents• Introduction of JVO SkyNode toolkit– used free software– architecture of JVO skynode• Problems in implementation and interoperability– XML  Java deserialization problem in AXIS
– Namespace problem ADQL, VOTable, STC– Usage of VOTable  id, name attributes ...
– Complexity of ADQL and STC object– ...

• Proposal– Simplify the ADQL and STC  Define minimum subset of ADQL andSTC and freeze them (never update, never change the namespace)– VOTable transfer  attachment or URL– Standardize the error message (not presented, as a future work)– ...

Development of the JVO SkyNode Toolkit• Primary aim:
– to provide a reference implementation for every kind ofdata service which uses ADQL & VOTable interface• Supported DBMS:– aimed to be independent on the type of DBMS
– The only requirement is availability of JDBC driver.
– but still PostgreSQL native SQL (copy command) is used...

• Restrictions:– Not all the ADQL syntax are supported.
– String representation of ADQL is JVOQL.

• Experimental Release:– http://jvo.nao.ac.jp/download/skynode-toolkit/

What can be done with the toolkit ?
• Catalog data query• Catalog data cross match query using
VOTable• Image data query• Image data cross match query using
VOTable• Spectrum data is not supported, but theframe work will be the same as that of
Catalog and Image.  next work
• You can build a sample SkyNode service.

Software used
• Tomcat 4.1.31 ---- servlet container• Axis 1.2RC1 ---- web service engine• J2SDK 1.4.2 ---- Java compiler & library• Ant 1.6.1 ---- Java-based build tool
• JavaCC 3.2 ---- parser generator for Java
• JAXB v1.0.3-b18-fcs ---- XML Java conversion• PostgreSQL 7.4.7 ---- DBMS
• Java HTM library (JHU) ---- spherical indexing• Java FITS library (HEASARC) ---- FITS IO lib• ...

ArchitectureJVO SkyNode Toolkit Architecture:QuerySkyNode

Client-Request
QuerySkyNode class
SQL
converts the client-request
SQL
executedon thebackend
DBMS
VOTableSQL to the SQL which is aquery actually executed onthe backend DBMS.
Service provider may extends
this class and override somemethods to configure for
their database.

: DataFormatter: FunctionColumn
User view table may have afunction type column whichis calculated according tothe query result of theother columns.

Column• name• dataType• unit• UCD• ...

ColumnMetadata
Table

Table
• column1• column2• ...

User View
This is a virtual table
exposed to the world.
This table defines what
data is available andwhich search parameters
can be specified.

regionTable

regionHTM• queryID• regionID• ra• dec
• size1• size2• regionExpr• queryID• regionID• htmLow• htmUpp

Real Tables

mainTable

mainHTM• dataID• ...

• dataID• htmIndex

This is acollection oftables actuallyqueried.

Space Region is approximately
expressed as a collection of
HTM index and the original
region condition is replaced by
the condition expressed in
HTM index ranges.

Catalog Data Query by ADQL
select ra, dec, magfrom catalogTable
where Point(ra, dec) within
Box ( (280, +20), 1.0, 1.0)
and mag < 24:QueryCatalog

catalogTable(1) Insert regiondata• objectID• ra• dec
• mag•…

User View(3) Create SQL for real tables
select *(2) Insert HTM
from (select * from regionTable queryID = 1) t1, (4) Execute
regionTable index for thenatural join regionHTM t2,Query• queryID
specified region
join catalogHTM t3• regionIDon t3.htm between t2.htmLow and t2.htmUpp,• ranatural join (select *• decfrom catalogTable• size1
where mag < 24) t4,• size2• regionExpr

regionHTM• queryID• regionID• htmLow• htmUpp

Real Table: DataFormatter

VOTable(6) Query Result inVOTable(5) Query Result

Column• name• dataType• unit• UCD• ...

ColumnMetadata
Table

catalogTable

catalogHTM• objectID• ra• dec
• mag•…

• objectID• htmIndex

Catalog Data Xmatch Query with VOTable
select vot.*, cat.*from EXT::0 vot, catalogTable catwhere distance((vot.ra, vot.dec),(cat.ra, cat.dec)) < 1 [arcsec]
and cat.mag < 24:QueryCatalogXmatch(1) Insert votabledata and region datacatalogTable• objectID• ra• dec
• mag•…

User View(3) Create SQL for real tables
select *(2) Insert HTM
from (select * from regionTable queryID = 1) t1, (4) Execute
regionTable index for thenatural join regionHTM t2,Query• queryID
specified region
natural join votable t3,• regionID
join catalogHTM t4
votable• ra
on t4.htm between t2.htmLow and t2.htmUpp,• dec
natural join (select *• regionID•size1from catalogTable• ra• size2
where mag < 24) t4,• dec• regionExpr•…

: DataFormatter

VOTable(6) Query Result in
VOTable(5) Query ResultColumn• name• dataType• unit
• UCD
• ...

ColumnMetadata
Table

catalogTable

catalogHTM• objectID• ra• dec
• mag•…

• objectID• htmIndex

regionHTM• queryID• regionID• htmLow• htmUpp

Real TableProblem encounteredin implementationand problem forinteroperability

Problems encountered in implementation (1)• XML

Java auto-conversion in AXIS

– With the standard usage of AXIS, creation of a Java
object corresponding to a XML document (e.g. VOTable)
is, as a default, required.
– Server memory is easily exhausted.
– Even several hundreds records of VOTable suffers fromout of memory error.

• Possible Solution :– Don’t use the auto-serialization and deserialization
mechanism of AXIS (suggestion from an AstroGridperson).
– return VOTable as an attachment– return a reference URL to retrieve the VOTableProblem in implementation (2)• Usage of VOTable is not clearly defined– id and name attribute  what should befilled ?
– Where column alias name should be put.
This information might be used for postsearch processing on portal side.
– Location where a table name and an alias
table name are put.
– Information on the origin of the column
data should be kept anywhere in VOTable.

Problems for interoperability (1)• Name space problem (as of 2005 Jan)– JVO  ADQL v0.8 + VOTable v1.1 + STC v1.1
– NVO  ADQL v0.74 + VOTable <v1.0 + NVO-STC• Temporal workaround– External interface  ADQL v0.74, VOTable v1.0
– Internal interface  ADQL v0.8, VOTable v1.1
– Namespace exchanger• Complexity of ADQL and STC object– ADQL  33 elements, 69 types– STC  250 elements, 88 types• Possible Solutions:– Define a core part of ADQL as a minimum subset andassign a permanent namespace. Never update, neverchange the namespace of the core part.

Minimum subset of ADQL
Element: 33 (full)  12 (basic)
Simple Type: 13 (f)  4 (b)Fundamentral Type

Simple Typexs:unsignedInt
xs:string(*)
xs:double(*)
xs:long(*)

aggregateFunctionNameType
allOrDistinctTypebinaryOperatorTypecomparisonType(*)
jointTableQualifierTypemathFunctionNameTypeorderDirectionType
trigonometricFunctionNameType
unaryOperatorType

Complex Type: 56 (f)  12 (b)ElementAllowArg(*)
ColumnCondition(*)
EndCommentExpression(*)
From(*)
GroupBy
HavingInToItem(*)
Literal(*)
Name
NatureOrder
OrderBy
ParamsPattern
QualifierRegion(*)

Complex Type
Restrict
Select(*)
SelectionList(*)
Set
Sigma
StartComment
Table(*)
TableName
TablesUnit(*)
Where(*)
fromTableType
selection

includeTableType
ArrayOfFromTableType
inclusionSetType
ConstantListSet
inclusiveSearchType
aggregateFunctionTypealiasSelectionItemType(*) integerType(*)
intersectionSearchType(*)
allSelectionItemType
intoType
archiveTableType
inverseSearchType
atomType(*)
joinTableTypebetweenPredType
likePredTypebinaryExprTypeliteralType(*)
closedExprTypemathFunctionType
closedSearchType
notBetweenPredTypecolumnReferenceType(*)
notLikePredTypecomparisonPredType(*)
numberTypedropTableTypeorderExpressionType
exclusiveSearchTypeorderOptionType
fromTableTypeorderType
fromType(*)
realType(*)
functionType
regionSearchTypegroupByType
scalarExpressionType
havingTypesearchType
selectType
selectionItemType
selectionLimitType
selectionListType
selectionOptionTypestringType(*)
subQuerySet
tableType(*)
trigonometricFunctionType
unaryExprTypeunionSearchType
userDefinedFunctionType
whereType(*)
xMatchTableAliasTypexMatchTyp

ComparisonPredType as a “must-support”
ConditionType• A basic SkyNode MUST support “comparisonPredType” .
• A basic SkyNode MAY support the other searchType.
• A basic SkyNode MUST recognize an unsupported
searchType as a “trueType”.
• A basic SkyNode MUST support the following construct:– <STC_columnName> <STC_operator> <STC_searchLocationType>– STC_operator ::= within, overlaps, outside

STC-type column must becompared with an object of
STC seachLocationType– point within STC(‘Circle ICRS 200 –20 2.0’ ) (Catalog Query)– region overlaps STC(‘Circle ICRS 200 –20 2.0’) (Image Query)– spectrum overlaps STC(‘SpectralInterval A 4000 7000’) andobservationTime within STC(‘TimeInterval 2004-05-01 2004-05utype defines31’)(Specturm Query)which column
represents STC.

ADQL-x version ofpoint within ‘Circle ICRS 200 –20 2.0’

It might be convenient todefine substitution types
for frequently used framessuch as <SpaceFrame
xsi:tye=“ICRSFrameType”/><Condition comparison=“within” xsi:type=“comparisonPredType"><Arg Table=“t” utype=“src.position” xsi:type=“columnReferenceType”/><Arg xsi:type=“searchLocationType"><AstroCoordSystem ID=“ICRS”><SpaceFrame><ICRS/><BARYCENTER/><SPHERICAL coord_naxes=“2”/></SpaceFrame></AstroCoordSystem><AstroCoordArea ID=“SearchRegion” coord_system_id=“ICRS”><Region><Circle unit=“deg”> <Center>200 –20</Center> <Radius>2.0</Radius></Circle></Region></AstroCoordArea>
Which frame should be defined as</Arg>
a “must be supported” frame</Condition>

Problem for interoperability (2)• Column name must be known in advance for writingADQL.
• We can get column names by “Columns” interface
and write ADQL, but it requires human intervention.
• A possible solution:– use UCD or Utype for specifying a column– Introduce “ucd” and “utype” attributes to the columnReferenceType<Item xsi:type=“columnReferenceType” Name=“ra” Table=“qso”/><Item xsi:type=“columnReferenceType” ucd=“pos.eq;src” Table=“qso”/><Item xsi:type=“columnReferenceType” utype=“Target.pos” Table=“qso”/>– If the specified utype or ucd is not found in the queried table,
 ignore the condition for that column return PARAMETER of “NaN” for that column

Summary• Experimental release of JVO SkyNodetoolkit– http://jvo.nao.ac.jp/download/skynode-toolkit/– Support for Catalog query, Image query• Some Proposals– Need minimum subset of ADQL and STC– Minor update on ADQL: ucd and utype attributes
to the ColumnReferenceType.
– Usage of VOTable. Location where column name,column alias name, table name and table alias
name are described.
– Error message (for future work)