From dishfits-request@fits.CX.NRAO.EDU Wed Oct  4 13:26:17 1989
Received: from NRAO.EDU (cv3.cv.nrao.EDU) by fits.cx.nrao.edu (4.0/SMI-DDN)
	id AA02422; Wed, 4 Oct 89 13:26:15 EDT
Received: from fits.cx.nrao.edu by NRAO.EDU (3.2/DCW-2n    )
	id AA12945; Wed, 4 Oct 89 13:27:18 EDT
Received: by fits.cx.nrao.edu (4.0/SMI-DDN)
	id AA02418; Wed, 4 Oct 89 13:18:59 EDT
Return-Path: <HLISZT@cvax.CV.NRAO.EDU>
X-Vms-Mail-To: DISHFITS
Message-Id: <891004130929.0000214A0B1@cvax.CV.NRAO.EDU>
Status: RO
From: HLISZT@NRAO.EDU
Sender: dishfits-request@fits.CX.NRAO.EDU
To: dishfits@fits.CX.NRAO.EDU
Subject: Draft agenda and travel info for single-dish meeting in Green Bank
Date: Wed,  4 Oct 89 13:09:29 EDT
Single-Dish FITS Standards Workshop Green Bank W. Va, 31 October-1 November 1989
Dear Colleague:
  Thank you for the interest you have shown in establishing a standard data
exchange format for single-dish radioastronomy.  With interest and enthusiasm,
we will create one!
  This message contains travel information and a reply form which you can return
via post or EMail. We are also sending by conventional mail a package which is
similar to this one but with more complete travel information (and a map).
  The last portion of this mail message contains a draft agenda for the
meeting.  You will notice that this agenda begins with a formal schedule of
presentations concerning various aspects of FITS, but soon becomes only an
outline of steps which we might take to design and implement the new standard.
Any suggestions you might have regarding the format of the meeting are welcome.
If you have (even preliminary) ideas concerning data exchange, please let us
hear from you; the floor will be open in what we hope will be a free and
spirited exchange of such ideas.  If you wish to make a formal presentation,
"we" would be glad to place you on the program.
  On the second day of the meeting, we have placed on the agenda a rather
general item concerning single-dish software.  If the previous discussions
fill the available time, this item will be sacrificed.  However, if time is
available, it would be informative for the group to hear about recent
developments at your telescope.
================================================================================
      INFORMATION REGARDING TRAVEL TO GREEN BANK (AND/OR CHARLOTTESVILLE)
  Transportation will largely be the responsibility of the individual
participants.  The nearest major airports to Green Bank are in Washington, DC
[Dulles (IAD) and National (DCA)] and Pittsburgh, PA.  Direct travel to GB
from either of these metropolitan areas involves a 4-5 hour drive through the
picturesque and twisty mountains of West Virginia and western VA. Otherwise,
one might wish to fly into Charlottesville, VA where transport between NRAO
Headquarters and Green Bank will be arranged. Green Bank is about 75 miles due
West of Charlottesville, or about 120 miles (some 3 hours) by road.  About one
hour of this trip is fairly adventurous driving. A map giving driving directions
to Green Bank is included in the non-electronic version of this mailing.
  We regret that there can be no financial subsidies from the NRAO. However,
once in Green Bank, lodging in the NRAO dormitories and on-site houses
(this may be either double or single occupancy) will be provided without charge.
The NRAO will also provide meals and frequent liquid refreshment.  For those
specifically desiring single rooms, lodging in nearby motels can be arranged
(by the NRAO) at the participant's expense.  Carpooling or van pickup for the
sessions will be organized.
  The weather at the end of October will be fairly cold with crisp nights
(Green Bank is at 2700 ft altitude).  Visitors are advised to bring warm
clothes and to be prepared for the possibility of rain or snow.
================================================================================
                           REGARDING YOUR TRAVEL PLANS
Please return by post to
                H. Liszt/Single-Dish Fits Meeting
                NRAO
                Edgemont Road
                Charlottesville, VA USA 22903-2475,
 or EMail via
   BITNET to HLISZT@NRAO, Internet to HLISZT@NRAO.edu, SPAN to 6654::HLISZT
 or phone
  Harvey Liszt 804-296-0344 (an answering machine will pick up the phone after
  3 1/2 rings)
--------------------------------------------------------------------------------
Name:...............................   Phone/E-Mail:............................
I will arrive in (city)....................... at (date/time)...................
And will depart from.......................... at (date/time)...................
Before the meeting, I will need accomodations in Charlottesville on
  (dates)...............................
After the meeting, I will need accomodations in Charlottesville on
  (dates)...............................
Please arrange transport for me from Charlottesville to Green Bank on 30 Oct.
  yes....... no.......
Please arrange transport for me from Green Bank to Charlottesville on 1 Nov.
  yes....... no.......
I will share a room with......................................
I really strongly prefer a single room
  yes....... no.......
I will be accompanied by (non-participants only)........................
================================================================================
            Single-Dish FITS Standards Workshop--(DRAFT) AGENDA
-------------------- day 1 -- TUESDAY, OCTOBER 31, 1989 ------------------------
==>Introduction: (8:45 am)
        Welcome and Introductory Remarks.......................H. Liszt
        History, Background, and Current Status of FITS
                Origins and Evolution of FITS..................D. Wells
                Remarks from the AAS WGAS......................R. Hanisch
                The IAU FITS Standards Committee...............P. Grosbol
                The NASA FITS Standards Office.................B. Schlesinger
        Use of FITS in High-Energy Astrophysics................S. Murray
(Coffee)
==>Steps Toward a Single-Dish FITS Format:
        The 1986 Format........................................R. Maddelena
        Alternative: A Compact Binary Table Format.............D. Wells
        Other alternatives?....................................(Anyone)
        General Discussion of Design and Methodology............EVERYONE!
(Lunch)
==>Specifying A Standard For Single Dish Data Exchange:
        Methodology/Architecture/Design
                Use of tables, random groups, other FITS mechanisms
                Vocabulary--Variable types; Usage of keywords/table headers
                Semantics--Meaning of keywords, definition of units
                World Coordinates--describing the velocity (or frequency) axis
(Coffee)
==>Charting the Future, Part 1:
        For the immediate future
                Defining the crucial details
                Defining a working group to settle these details
                Construction of first-generation FITS readers and writers
(Cocktails and Dinner)
================================================================================
            Single-Dish FITS Standards Workshop--(DRAFT) AGENDA
------------------- day 2 -- WEDNESDAY, NOVEMBER 1, 1989------------------------
==>Charting the Future, Part 2: (8:45 am)
        For the long-term
                Disseminating the products of the working group
                General implementation of the new standard
                Assuring general acceptance of the standard
                Assuring adherence to the new standard
(coffee)
==>General Concerns Affecting the Software Performance of Single Dishes
        Presentations from various instruments.................?
==>Wrap-up And General Summary of the Workshop
================================================================================
From dishfits-request@fits.CX.NRAO.EDU Thu Oct  5 18:02:59 1989
Received: from NRAO.EDU (cv3.cv.nrao.EDU) by fits.cx.nrao.edu (4.0/SMI-DDN)
	id AA00824; Thu, 5 Oct 89 18:02:57 EDT
Received: from fits.cx.nrao.edu by NRAO.EDU (3.2/DCW-2n    )
	id AA16314; Thu, 5 Oct 89 18:03:23 EDT
Received: by fits.cx.nrao.edu (4.0/SMI-DDN)
	id AA00743; Thu, 5 Oct 89 17:55:48 EDT
Return-Path: <hanisch@stsci.edu>
Message-Id: <8910052150.AA13865@RA.STSCI.EDU>
Status: RO
From: (Bob Hanisch) <hanisch@stsci.edu>
Sender: dishfits-request@fits.CX.NRAO.EDU
To: dishfits@nrao.edu
Subject: single dish data and FITS format
Date: Thu, 5 Oct 89 16:50:51 EST
                Some Comments About Single-Dish FITS Format
                        and Related FITS Issues
                          Bob Hanisch, ST ScI
                              October 1989
  In preparation for the meeting in Green Bank later this month, I have
written down some ideas concerning how to handle the special case of single
dish radio astronomy data within the FITS format.  This is being distributed
in advance of the meeting so that attendees can think about the issues prior
to arrival.
  Harvey Liszt distributed an e-mail message to all of us which contained a
careful analysis of the header structures proposed by Stobie and Morgan.
Unfortunately I think this analysis does not really address the general
problem, i.e., how to manage the keyword space unique to a certain class of
instruments, detectors, or a particular project.  From the FITS point of view
this is the critical issue, and most of the items discussed in the memo are the
details that need to be worked out once the more general problem has been
addressed.
  What Stobie and Morgan described in their original memo is a simple
hierarchical keyword space.  The presence of the hierarchy is denoted by the
use of the SINGLDSH keyword, followed by a secondary hierarchy (GENERAL,
NRAO-12M, etc.) which further defines the scope of the keyword definitions.
The AIPS system has already implemented a hierarchical scheme of sorts, in
that AIPS uses the FITS COMMENT and HISTORY records for storing information
unique to AIPS.  Non-AIPS FITS readers have no obligation to parse the
contents of these cards beyond simply recording them with the other header
information, or even ignoring them if they wish.  The point here is to have a
keyword space which can be hidden from FITS reading programs that do not
understand or have no interest in some specialized set of keywords.  The
advantages of such a scheme include the fact that we need not have any
community-wide consensus on the definitions of ALL FITS keywords.  Rather,
subgroups such as the single-dish radio astronomy community can agree upon
their own conventions and be largely independent of other subdisciplines.
  If we agree that having some sort of hierarchical keyword naming scheme is
worthwhile (and I think that this is NOT a foregone conclusion -- more comments
on this later), the first thing to do is determine the best syntax for its
implementation.  I see several difficulties with the proposal of Stobie and
Margon (and implicitly with the AIPS technique of hiding similar information
behind COMMENT and HISTORY records):
o  The proposal is limited to a two-level hierarchy, a primary domain (SINGLDSH)
and a secondary domain (GENERAL, NRAO-12M, etc.).  This seems unnecessarily
restrictive, although it is clearly simple to generalize to several more levels.
o  The keyword hierarchy takes up a lot of space on the header line, leaving
little space for values or comments.  This situation is exacerbated if deeper
hierarchies are to be permitted.
o  FITS readers require special code to parse the header and retrieve the
data values.
o  Everytime a new domain is introduced, FITS readers everywhere will have to
be updated to recognize (and most often ignore) the new domain name.
  Some time ago I proposed implementing hierarchical keywords by REQUIRING that
each hierarchical declaration actually begin with the keyword HIERARCH (this was
during a FITS planning meeting held in Charlottesville in Jan. 1988, as I
recall).  The reasoning was that this is such an ugly keyword that no one else
in their right mind would use it, thus it would serve as a flag to indicate
that a special parser would have to go into action in order to retrieve the
actual keyword name and value buried further to the right on the card.  In
retrospect, this plan suffers the same disadvantages mentioned above EXCEPT for
the last one; by reserving a special keyword to introduce a hierarchical
keyword domain, the problem of the arbitrary introduction of new domains is
circumvented. However, the other problems, especially the lack of space on the
header card, are not resolved.
  An alternative scheme has come to mind (based on an idea proposed by Doug
Tody at the same meeting) which looks much more like simple nested do-loops in
Fortran, or structure definition statements in any modern computer language.
It would be based on begin/end statements for various keyword domains, and
could be nested arbitrarily deep, accommodating things like
SINGLDSH/GENERAL/NRAO/12METER/... or whatever.  I propose to use keywords of
the form KHBEGIN and KHEND to signify a keyword domain (KH = keyword hierarchy;
I don't really care what the particular names are so long as we can agree on
something unique and representative).  For example, the syntax might be
something like this (paying little attention to the actual column spacing):
From dishfits-request@fits.CX.NRAO.EDU Mon Oct  9 17:01:11 1989
Received: from NRAO.EDU (cv3.cv.nrao.EDU) by fits.cx.nrao.edu (4.0/SMI-DDN)
	id AA05146; Mon, 9 Oct 89 17:01:08 EDT
Received: from fits.cx.nrao.edu by NRAO.EDU (3.2/DCW-2n    )
	id AA26636; Mon, 9 Oct 89 17:02:06 EDT
Received: by fits.cx.nrao.edu (4.0/SMI-DDN)
	id AA05123; Mon, 9 Oct 89 16:55:43 EDT
Return-Path: <HLISZT@cvax.CV.NRAO.EDU>
X-Vms-Mail-To: DISHFITS
Message-Id: <891009165133.00004A45081@cvax.CV.NRAO.EDU>
Status: RO
From: HLISZT@NRAO.EDU
Sender: dishfits-request@fits.CX.NRAO.EDU
To: dishfits@fits.CX.NRAO.EDU
Subject: Comments by R. Hanisch (St Sci)
Date: Mon,  9 Oct 89 16:51:33 EDT
   ******************************************************************
              The following was sent to NRAO by R. Hanisch
               (hanisch@stsci.edu or hanisch@scivax.span)
   ******************************************************************
From:   STSCIC::HANISCH       5-OCT-1989 17:50:34.74
To:     STOBIE
CC:
Subj:   current version of FITS memo
                Some Comments About Single-Dish FITS Format
                        and Related FITS Issues
                          Bob Hanisch, ST ScI
                              October 1989
  In preparation for the meeting in Green Bank later this month, I have
written down some ideas concerning how to handle the special case of single
dish radio astronomy data within the FITS format.  This is being distributed
in advance of the meeting so that attendees can think about the issues prior
to arrival.
  Harvey Liszt distributed an e-mail message to all of us which contained a
careful analysis of the header structures proposed by Stobie and Morgan.
Unfortunately I think this analysis does not really address the general
problem, i.e., how to manage the keyword space unique to a certain class of
instruments, detectors, or a particular project.  From the FITS point of view
this is the critical issue, and most of the items discussed in the memo are the
details that need to be worked out once the more general problem has been
addressed.
  What Stobie and Morgan described in their original memo is a simple
hierarchical keyword space.  The presence of the hierarchy is denoted by the
use of the SINGLDSH keyword, followed by a secondary hierarchy (GENERAL,
NRAO-12M, etc.) which further defines the scope of the keyword definitions.
The AIPS system has already implemented a hierarchical scheme of sorts, in
that AIPS uses the FITS COMMENT and HISTORY records for storing information
unique to AIPS.  Non-AIPS FITS readers have no obligation to parse the
contents of these cards beyond simply recording them with the other header
information, or even ignoring them if they wish.  The point here is to have a
keyword space which can be hidden from FITS reading programs that do not
understand or have no interest in some specialized set of keywords.  The
advantages of such a scheme include the fact that we need not have any
community-wide consensus on the definitions of ALL FITS keywords.  Rather,
subgroups such as the single-dish radio astronomy community can agree upon
their own conventions and be largely independent of other subdisciplines.
  If we agree that having some sort of hierarchical keyword naming scheme is
worthwhile (and I think that this is NOT a foregone conclusion -- more comments
on this later), the first thing to do is determine the best syntax for its
implementation.  I see several difficulties with the proposal of Stobie and
Margon (and implicitly with the AIPS technique of hiding similar information
behind COMMENT and HISTORY records):
o  The proposal is limited to a two-level hierarchy, a primary domain (SINGLDSH)
and a secondary domain (GENERAL, NRAO-12M, etc.).  This seems unnecessarily
restrictive, although it is clearly simple to generalize to several more levels.
o  The keyword hierarchy takes up a lot of space on the header line, leaving
little space for values or comments.  This situation is exacerbated if deeper
hierarchies are to be permitted.
o  FITS readers require special code to parse the header and retrieve the
data values.
o  Everytime a new domain is introduced, FITS readers everywhere will have to
be updated to recognize (and most often ignore) the new domain name.
  Some time ago I proposed implementing hierarchical keywords by REQUIRING that
each hierarchical declaration actually begin with the keyword HIERARCH (this was
during a FITS planning meeting held in Charlottesville in Jan. 1988, as I
recall).  The reasoning was that this is such an ugly keyword that no one else
in their right mind would use it, thus it would serve as a flag to indicate
that a special parser would have to go into action in order to retrieve the
actual keyword name and value buried further to the right on the card.  In
retrospect, this plan suffers the same disadvantages mentioned above EXCEPT for
the last one; by reserving a special keyword to introduce a hierarchical
keyword domain, the problem of the arbitrary introduction of new domains is
circumvented. However, the other problems, especially the lack of space on the
header card, are not resolved.
  An alternative scheme has come to mind (based on an idea proposed by Doug
Tody at the same meeting) which looks much more like simple nested do-loops in
Fortran, or structure definition statements in any modern computer language.
It would be based on begin/end statements for various keyword domains, and
could be nested arbitrarily deep, accommodating things like
SINGLDSH/GENERAL/NRAO/12METER/... or whatever.  I propose to use keywords of
the form KHBEGIN and KHEND to signify a keyword domain (KH = keyword hierarchy;
I don't really care what the particular names are so long as we can agree on
something unique and representative).  For example, the syntax might be
something like this (paying little attention to the actual column spacing):
From dwells@fits.CX.NRAO.EDU Wed Oct 11 17:56:50 1989
Received: from NRAO.EDU (cv3.cv.nrao.EDU) by fits.cx.nrao.edu (4.0/SMI-DDN)
	id AA08854; Wed, 11 Oct 89 17:56:48 EDT
Received: from fits.cx.nrao.edu by NRAO.EDU (3.2/DCW-2n    )
	id AA02170; Wed, 11 Oct 89 17:56:20 EDT
Received: by fits.cx.nrao.edu (4.0/SMI-DDN)
	id AA08809; Wed, 11 Oct 89 17:38:09 EDT
Return-Path: <dwells>
Message-Id: <8910112138.AA08803@fits.cx.nrao.edu>
Status: RO
From: dwells@fits.CX.NRAO.EDU (Don Wells)
Sender: dishfits-request@fits.CX.NRAO.EDU
To: dishfits@fits.CX.NRAO.EDU
Subject: Re:  single dish data and FITS format
Date: Wed, 11 Oct 89 17:38:05 EDT
	From hanisch@stsci.edu Fri Oct  6 11:24:52 1989
	Subject: my memo seems to have been chopped off for some people
Don,
  My memo on hierarchical keywords seems to have been truncated for some
recipients.  Bill Cotton reported to me that his copy had been chopped, and
I just checked mine and it also is chopped, at the same place.  Could I ask
you to check on the mail exploder and see if it might be the culprit?  Here's
the text.  Please send it out to the exploder, if you would.  Have to head off
to the airport in the next few minutes.  Thanks, Bob.
[see technical explanation at end -- DWells]
                Some Comments About Single-Dish FITS Format
                        and Related FITS Issues
                          Bob Hanisch, ST ScI
                              October 1989
  In preparation for the meeting in Green Bank later this month, I have
written down some ideas concerning how to handle the special case of single
dish radio astronomy data within the FITS format.  This is being distributed
in advance of the meeting so that attendees can think about the issues prior
to arrival.
  Harvey Liszt distributed an e-mail message to all of us which contained a
careful analysis of the header structures proposed by Stobie and Morgan.
Unfortunately I think this analysis does not really address the general
problem, i.e., how to manage the keyword space unique to a certain class of
instruments, detectors, or a particular project.  From the FITS point of view
this is the critical issue, and most of the items discussed in the memo are the
details that need to be worked out once the more general problem has been
addressed.
  What Stobie and Morgan described in their original memo is a simple
hierarchical keyword space.  The presence of the hierarchy is denoted by the
use of the SINGLDSH keyword, followed by a secondary hierarchy (GENERAL,
NRAO-12M, etc.) which further defines the scope of the keyword definitions.
The AIPS system has already implemented a hierarchical scheme of sorts, in
that AIPS uses the FITS COMMENT and HISTORY records for storing information
unique to AIPS.  Non-AIPS FITS readers have no obligation to parse the
contents of these cards beyond simply recording them with the other header
information, or even ignoring them if they wish.  The point here is to have a
keyword space which can be hidden from FITS reading programs that do not
understand or have no interest in some specialized set of keywords.  The
advantages of such a scheme include the fact that we need not have any
community-wide consensus on the definitions of ALL FITS keywords.  Rather,
subgroups such as the single-dish radio astronomy community can agree upon
their own conventions and be largely independent of other subdisciplines.
  If we agree that having some sort of hierarchical keyword naming scheme is
worthwhile (and I think that this is NOT a foregone conclusion -- more comments
on this later), the first thing to do is determine the best syntax for its
implementation.  I see several difficulties with the proposal of Stobie and
Margon (and implicitly with the AIPS technique of hiding similar information
behind COMMENT and HISTORY records):
o  The proposal is limited to a two-level hierarchy, a primary domain (SINGLDSH)
and a secondary domain (GENERAL, NRAO-12M, etc.).  This seems unnecessarily
restrictive, although it is clearly simple to generalize to several more levels.
o  The keyword hierarchy takes up a lot of space on the header line, leaving
little space for values or comments.  This situation is exacerbated if deeper
hierarchies are to be permitted.
o  FITS readers require special code to parse the header and retrieve the
data values.
o  Everytime a new domain is introduced, FITS readers everywhere will have to
be updated to recognize (and most often ignore) the new domain name.
  Some time ago I proposed implementing hierarchical keywords by REQUIRING that
each hierarchical declaration actually begin with the keyword HIERARCH (this was
during a FITS planning meeting held in Charlottesville in Jan. 1988, as I
recall).  The reasoning was that this is such an ugly keyword that no one else
in their right mind would use it, thus it would serve as a flag to indicate
that a special parser would have to go into action in order to retrieve the
actual keyword name and value buried further to the right on the card.  In
retrospect, this plan suffers the same disadvantages mentioned above EXCEPT for
the last one; by reserving a special keyword to introduce a hierarchical
keyword domain, the problem of the arbitrary introduction of new domains is
circumvented. However, the other problems, especially the lack of space on the
header card, are not resolved.
  An alternative scheme has come to mind (based on an idea proposed by Doug
Tody at the same meeting) which looks much more like simple nested do-loops in
Fortran, or structure definition statements in any modern computer language.
It would be based on begin/end statements for various keyword domains, and
could be nested arbitrarily deep, accommodating things like
SINGLDSH/GENERAL/NRAO/12METER/... or whatever.  I propose to use keywords of
the form KHBEGIN and KHEND to signify a keyword domain (KH = keyword hierarchy;
I don't really care what the particular names are so long as we can agree on
something unique and representative).  For example, the syntax might be
something like this (paying little attention to the actual column spacing):
	.
	.  (standard FITS header)
	.
	.
KHBEGIN = 'SINGLDSH'		/ begin a keyword hierarchy called SINGLDSH
STATION = 'NRAO-12M'		/ one or more keywords unique to this
UNIQKW2 = 1.2345678		/   hierarachy
	.
	.
KHBEGIN = 'GENERAL'		/ second level of hierarchy
UNIQKW3 = 9.8765432		/ one or more keywords unique to this
	.				/   second level of the hierarchy
	.
	.
	.  (other KHBEGIN keywords define additional levels of the hierarchy,
	.   as needed)
KHEND   = 'GENERAL'		/ close out the GENERAL keyword list
KHEND   = 'SINGLDSH'		/ close out the SINGLDSH keyword list
	.
	.
Now, there are some things that I haven't thought through that carefully yet,
like whether or not the KHBEGIN and KHEND keywords should have a numeral
attached (minimizing the number of keywords with the same name and perhaps
making the parsing of the hierarchy a bit more fool-proof).  Also, there's the
question of whether you would want to allow the full flexibility of going in
and out of a particular domain.  That is, should this sort of thing be
permitted:?
KHBEGIN = 'SINGLDSH'		/ begin a keyword hierarchy called SINGLDSH
	.
	.
KHBEGIN = 'GENERAL'		/ second level of hierarchy
	.
	.
KHEND   = 'GENERAL'		/ close out the GENERAL keyword list
	.
	.
KHBEGIN = 'NRAO-12M'		/ another second level hierarchy
	.
	.
KHEND   = 'NRAO-12M'
	.
	.
KHBEGIN = 'GENERAL'		/ reopen the GENERAL list
	.
	.
KHEND   = 'GENERAL'
KHEND   = 'SINGLDSH'		/ close out the SINGLDSH keyword list
This is not a big deal, and maybe it would simply be a matter of style whether
or not this would be allowed or just discouraged.  However, there are a couple
of major advantages of this approach to managing the keyword space:
o  The keywords within the hierarchy can all be parsed with standard FITS
readers.
o  There is no wasted space on the individual keyword cards, allowing more
complete use of the comment field.  If there are enough `private' keywords to
warrant setting up a hierarchy, the overhead of the BEGIN and END cards is
small.
o  The logic for skipping over the `private' keyword definitions is
straightforward:  upon encountering a KHBEGIN card, the reader would be free
to skip parsing other cards until the corresponding KHEND card was found.
In order to simplify this somewhat, we may wish to limit the depth of the
hierarchy, say to 9 levels (that should be enough for just about everyone,
and would allow the formation of numbered hierarchies, KHBEGIN1, KHBEGIN2,
etc., as described above).  There is clearly redundant information in using
both the numbering scheme AND the value field of the BEGIN/END cards.
o  The definition of new keyword hierarchies (SINGLDSH, INTERF, AIPS, CCD,
HST, etc.) does not require FITS readers to be updated.  A one-time update
is all that is necessary, and even this is not essential for existing FITS
readers to work since they could simply ignore all of this stuff, with no
harm done.
  Thus, if the need for private keyword definition spaces is genuine, a
scheme such as that proposed here might be a more viable one for implementation
than that previously proposed.  However, the question remains as to whether
such private keyword domains are even required.  Just to review this situation
a bit, the prime motivation for having a hierarchical keyword space of some
sort is so that various projects, disciplines, etc., can define any set of new
keywords without having to worry about name collisions with other such groups.
But is this really a concern?  Does it really matter if a CCD data set and a
single-dish radio data set have inconsistent definitions of BIAS?  or REFPT?
or DELTAX?  I guess it depends on what level of interoperability we expect to
achieve with FITS.  In a practical sense I think we all recognize that the idea
of establishing some sort of global dictionary for FITS keywords is impossible.
The hierarchical scheme says, ok, we can't do it so let's hide all of our
private definitions.  The alternative says, ok, we can't do it but it doesn't
really matter because the types of data we're transferring are so heterogeneous.
  (By the way, I wish to point out something about the practice of defining
new FITS keywords.  There seems to have arisen some idea that it is forbidden
to define keywords other than those mentioned in the series of FITS standards
papers.  Nonsense.  Reread Paper I, p. 366, middle of the 2nd column:
    "Note that the two long keywords, INSTRUMEN and TELESCOP are names which
    have been truncated after 8 characters.  We suggest this as a good
    convention for the creation of new keywords."
Whether they intended it or not, Don, Eric, and Ron left the door wide open for
the creation of new keywords and even recommended how it should be done.)
  What does matter, I think, is to have a small set of reserved keywords that
all FITS readers might reasonably be expected to understand, or at least not
interfere with.  FITS Paper I defines the fundamental set (SIMPLE, BITPIX,
NAXIS, NAXISi (i=1,NAXIS), and END) and an optional set (BSCALE, BZERO,
BUNIT, CRVALi, etc.).  One solution to this problem would be for the various
FITS committees to review Paper I and agree upon such a set of reserved
keywords, and then let various projects use the rest of the keyword space to
their hearts' desires.  This is, after all, the situation we have now and
maybe it's not so bad.  Anyway, wait until you all see HST data!  In any case,
I think that a clarification and standardization of some small set of keywords
is desirable regardless of whether we adopt a hierarchical naming convention
project or discipline dependent keywords.
  Well, those are my thoughts at the moment.  I've gone on a bit longer than
I thought, but I hope that those of you who've stuck with me to this point
have got some food for thought and that we can have a lively discussion about
this in Green Bank.  See you all there.
======================= non-FITS technical discussion below: =============
Why did Bob's message on 05 October get truncated?
The SMTP (Simple Mail Transfer Protocol) of the Internet terminates a
message by a line that has *only* one period (.) on it. This rule allows
the mail protocol to work the same way on top of a variety of lower
protocols, character sets and OSes.  So, what does an SMTP mailer do if a
message contains a text line with only a period? Why, double it, of course!
And, of course, a mail daemon which receives such a message will change the
double period back to a single period before delivery to the customer
mailbox.
Messages containing single-period lines are rare. Bob's message at 1755EDT
last Thursday 05 October contains the first such lines I have seen in
several years. His message was truncated *exactly* at the first occurrence
of such a line, and so we know that the problem may be that some mail daemon
somewhere failed to double the period on transmission.
At present I am unable to say with certainty which of three different
computers, one at STScI and two at NRAO, was the guilty party.
I have added tabs before the periods in this copy to defeat the RFC822
rule.
-Don
From dishfits-request@fits.CX.NRAO.EDU Fri Oct 13 20:34:52 1989
Received: from NRAO.EDU (cv3.cv.nrao.EDU) by fits.cx.nrao.edu (4.0/SMI-DDN)
	id AA02034; Fri, 13 Oct 89 20:34:47 EDT
Received: from fits.cx.nrao.edu by NRAO.EDU (3.2/DCW-2n    )
	id AA02650; Fri, 13 Oct 89 20:34:18 EDT
Received: by fits.cx.nrao.edu (4.0/SMI-DDN)
	id AA02026; Fri, 13 Oct 89 20:08:14 EDT
Return-Path: <POSTMASTER@NRAO.EDU>
X-Vms-Mail-To: DISHFITS AT NRAO
Message-Id: <891013195606.00000B24031@cvax.CV.NRAO.EDU>
Status: RO
From: "GUILLOTE@FRGAG51" <POSTMASTER@NRAO.EDU>
Sender: dishfits-request@fits.CX.NRAO.EDU
To: dishfits@fits.CX.NRAO.EDU
Subject: Single-dish FITS
Date: Fri, 13 Oct 89 19:56:06 EDT
Date sent:  13-OCT-1989 13:52:50.84
To:   DISHFITS@NRAO
                Single-Dish FITS Format
        Stephane Guilloteau (1) and Thierry Forveille (2)
(1) Institut de Radio Astronomie Millimetrique (IRAM)
    300 Rue de la Piscine
        F-38406 Saint Martin d'Here     FRANCE
E-Mail  "GUILLOTE@FRGAG51.BITNET"
(2) Groupe d'Astrophysique de Grenoble          (GAG)
    Observatoire de Grenoble
        BP 53 X
        F-38041 Grenoble Cedex          FRANCE
E-Mail  "FORVEILL@FRGAG51.BITNET"
        Dear collegues,
In the previous mails delivered through the DISHFITS exploder,
the goal of using FITS for single-dish data was not clearly exposed.
We present here an (possibly obvious, and certainly not exhaustive)
analysis of the basic needs and goals, leading to a proposal.
FITS is designed for "transport" of images or other data. For
single-dish data, we may consider several cases of "transport"
0) Transport between the same analysis program, running on
   identical computers, on different sites. FITS is not required
   in this case
1) Transport between the same analysis program, running on
   different computers. FITS may be required, since the binary
   data format can be different on the two computers. Using
   FITS instead of dedicated conversion programs avoids the need
   of dedicated programs on each type of computer
2) Transport between different analysis programs, probably running
   on different computers.
3) As a subset of 2), possible transport between dedicated single-dish
   analysis programs and "general" image processing systems (AIPS or
   others).
4) Archiving. FITS may be desirable in this case, as it guarantees
   the readability on any computer later on.
Each case impose some constraint on the design of a single-dish
(or image) FITS format.
Case 1) and 4) require that ALL the information be written on FITS
tape. This can be handled using dedicated keywords, or, as done for
AIPS, substructures in the COMMENT and HISTORY cards. Whatever the
choice of implementation, keywords are required matching EXACTLY the
data format, with a simple provision for future extensions.
IRAM and other observatories are currently using the CLASS program for
single-dish data analysis. The data structure for this program is
segmented in "Sections" describing some aspects of the data. The
currently used sections is described in Appendix in this MAIL.
Case 2), considered here as exchange between two single-dish data
analysis programs, requires some agreement upon a reasonable subset of
keywords in order to preserve most (if not all) information in the
transport. Some loss of information seems unavoidable, since not all
analysis programs have a one to one correspondence in header
parameters.
Case 3) is even more tricky. It is likely that a lot of information
will be lost in this case. However, a minimal requirement should be
that the data can be correctly transferred. Following the original
FITS rule, this would imply putting the data in the standard data
area, NOT in a table extension. The minimal subset of keywords to be
defined would then be NAXIS, NAXISi and BITPIX. However, most image
processing are (or should be) able to process coordinates properly,
including frequency information, and it would be highly desirable to
transfer in the most general way this information too. Looking a
little bit closer to what is a single-dish data set, we may consider
two cases :
        - Spectra
        - Continuum drifts
Spectra can be considered as 1-dimensional subsets of 3-D data cubes.
AIPS-style FITS headers could be readily used to described completely
REDUCED single-dish spectra. As an example, consider the following
FITS header, which has been produced by the CLASS to FITS conversion
program :
----------------------------------------------------------------------
SIMPLE  =                    T         /
BITPIX  =                   16         /
NAXIS   =                    4         /
NAXIS1  =                  512         /
NAXIS2  =                    1         /
NAXIS3  =                    1         /
NAXIS4  =                    1         /
BLANK   =                32767         / Blanking value
BSCALE  =  0.1595546564204E-05         /
BZERO   =  0.4077937081456E-01         /
DATAMIN = -0.1150350086391E-01         /
DATAMAX =  0.9305904805660E-01         /
BLOCKED =                    T         /
BUNIT   = 'K       '                   /
CTYPE1  = 'FREQ    '                   /
CRVAL1  =  0.0000000000000E+00         / Offset frequency
CDELT1  =  0.1220703125000E+05         / Frequency resolution
CRPIX1  =  0.2565000000000E+03         /
CTYPE2  = 'RA---GLS'                   /
EQUINOX =  0.1950000000000E+04         /
CRVAL2  =  0.5150368630886E+02         /
CDELT2  =  0.8333333046443E-01         /
CRPIX2  =  0.0000000000000E+00         /
CTYPE3  = 'DEC--GLS'                   /
CRVAL3  =  0.3108135491610E+02         /
CDELT3  =  0.8333333046443E-01         /
CRPIX3  =  0.0000000000000E+00         /
CTYPE4  = 'STOKES  '                   /
CRVAL4  =      1.0000000000000         /
CDELT4  =      0.0000000000000         /
CRPIX4  =      0.0000000000000         /
TELESCOP= '100-M       '               /
OBJECT  = 'NGC1333     '               /
LINE    = 'NH3(1,1)    '               / Line name
RESTFREQ=  0.2369449499989E+11         / Rest frequency
VLSR    =  0.6500000000000E+04         / Velocity of reference channel
ORIGIN  = 'CLASS-Grenoble  '           /
DATE    = '13/10/89'                   / Date written
DATE-OBS= ' 1/ 9/84'                   / Date observed
DATE-RED= '27/ 6/89'                   / Date reduced
END
----------------------------------------------------------------------
Note that CRVAL2 and CRVAL3 are coordinates at the point of tangency,
as should be, and CDELT2 CDELT3 represent here the offsets of the
observed point with respect to this reference direction. Keywords
CD00i00i may be used instead. The frequency axis is defined as an
offset to a rest frequency given in keyword RESTFREQ, and the
corresponding source velocity is given in keyword VLSR. (It may be
better to specify the frequency axis as effective observing frequency,
and use a keyword VELO-LSR for the source velocity). This header is
very close to a standard AIPS FITS header produced for a 3-D VLM cube
for example, and no "vital" information has been lost if the data is
already reduced.
We are not expert in "image" processing, but presumably such a FITS
header could be understood by any FITS reading program without major
loss of information (the only two non standard keywords are RESTFREQ
and VLSR).
Continuum drifts are handled in a similar way: the frequency axis has
dimension 1, and one sky coordinate axis has dimension equal to the
number of data points. It may be interesting to define  a very special
"projection code" for drifts in horizontal coordinates, which are
frequently used to check the telescope pointing, e.g. something like
-HOR.
Turning back to points 1) and 4), which imply in fact essentially the
same constraints, we believe that a hierarchical structure as
described by Bob Hanisch essentially fulfills all the requirements. As
an example, here is a (quasi) FITS header part which corresponds to
the data structure used in CLASS (See Appendix). Reference to notes
are numbered (*i). We did not bother editing a real FITS header, but
only the keywords are mentionned.
KHBEGIN = 'CLASS'                       / (*1)
VERSION = 'SEP-1989'                    / Date of version
KHBEGIN = 'POINTER'                     / (*2)
GENERAL = 20                            /
POSITION= 11                            /
SPECTRO = 15                            / Spectroscopic data
FSWITCH = 5                             / for NPHASE = 1
DRIFT   = 0                             / Not present
BEAM    = 0
CALIBRAT= 19                            / May be shorter
SKYDIP  = 0
GAUSS   = 9                             / for NLINE = 1
HFS     = 0
SHELL   = 0
CONTINUU= 0
HISTORY = 9                             / Sequence = 4
PLOT    = 4
BASE    = 8                             / For 2 windows
XCOORD  = 0                             / (*5)
KHEND
COMMENT * OBS.GENERAL : General parameters, always present
KHBEGIN ='GENERAL'      ! (mgen=9,mgen2=11)
NUMBER  =               ! rnum      ! Observation number
VERSION =               ! rver      ! Version number
TELESCOP=               ! rteles(3) ! Telescope name
DATE_OBS=               ! rdobs     ! Date of observation
DATE_RED=               ! rdred     ! Date of reduction
SYSTEM  =               ! rtypec    ! Type of coordinates
KIND    =               ! rkind     ! Type of data
QUALITY =               ! rqual     ! Quality of data
SCAN    =               ! rscan     ! Scan number
UT_TIME =               ! rut       ! UT of observation
LST_TIME=               ! rst       ! LST of observation
AZIMUTH =               ! raz       ! Azimuth
ELEVATIO=               ! rel       ! Elevation
TAU     =               ! rtau      ! Opacity
TSYS    =               ! rtsys     ! System temperature
INTEGRAT=               ! rtime     ! Integration time
KHEND
COMMENT * OBS.POSITION : Position information.
KHBEGIN ='POSITION'     ! (mpos=11)
SOURCE  =               ! rsourc(3) ! Source name
EPOCH   =               ! repoch    ! Epoch of coordinates
LAMBDA  =               ! rlam      ! Lambda
BETA    =               ! rbet      ! Beta
LAMBDA_O=               ! rlamof    ! Offset in Lambda
BETA_OFF=               ! rbetof    ! Offset in Beta
PROJECTI=               ! rproj     ! Projection system
KHEND
COMMENT * OBS.SPECTRO : Spectroscopic information (for spectra)
KHBEGIN ='SPECTRO'      ! (mspec=15)
LINE    =               ! rline(3)  ! Line name
REST    =               ! rrestf    ! Rest frequency
NCHAN   =               ! rnchan    ! Number of channels
REFERENC=               ! rrchan    ! Reference channel
FRES    =               ! rfres     ! Frequency resolution
FOFF    =               ! rfoff     ! Frequency offset
VRES    =               ! rvres     ! Velocity resolution
VOFF    =               ! rvoff     ! Velocity at reference channel
BAD     =               ! rbad      ! Blanking value
IMAGE   =               ! rimage    ! Image frequency
VELOCITY=               ! rvtype    ! Type of velocity
KHEND
COMMENT * OBS.BASE : Baseline information (for spectra of drifts)
KHBEGIN ='BASE'         ! (mwind=20,mbase=4+2*mwind)
DEGREE  =               ! rdeg      ! Degree of last baseline
SIGMA   =               ! rsigfi    ! Sigma
AREA    =               ! raire     ! Area under windows
NWINDOW =               ! rnwind    ! Number of line windows
WIND1(NWINDOW)          ! rw1(mwind)! Lower limits of windows (*3)
WIND2(NWINDOW)          ! rw2(mwind)! Upper limits of windows
KHEND
COMMENT * OBS.HISTORY : Scan numbers of initial observations
KHBEGIN ='HISTORY'      ! (mseq=100,morig=2*mseq+1)
SEQUENCE=               ! rnseq     ! Number of sequences
START(SEQUENCE)         ! rstart(mseq) ! Start can number of seq. (*4)
REND(SEQUENCE)          ! rend(mseq)   ! End scan number of seq.
KHEND
COMMENT * OBS.PLOT : Default plotting limits.
KHBEGIN = 'PLOT'                ! (mplot=4)
YMIN    =               ! ramin     ! Min Y value plotted
YMAX    =               ! ramax     ! Max Y value plotted
XMIN    =               ! rvmin     ! Min X value plotted
XMAX    =               ! rvmax     ! Max X value plotted
KHEND
COMMENT * OBS.CALIBRATION : Calibration parameters
KHBEGIN = 'CALIBRATION' ! (mcalib=19)
BEAM_EFF=               ! rbeeff    ! Beam efficiency
FORW_EFF=               ! rfoeff    ! Forward efficiency
GAIN_IM =               ! rgaini    ! Image/Signal gain ratio
H2OMM   =               ! rh2omm    ! MM of water vapor
PAMB    =               ! rpamb     ! Ambient pressure (hPa)
TAMB    =               ! rtamb     ! Ambient temperature (K)
TATMSIG =               ! rtatms    ! Atmosphere temp. signal band
TCHOP   =               ! rtchop    ! Chopper temperature
TCOLD   =               ! rtcold    ! Cold load temperature
TAUSIG  =               ! rtaus     ! Opacity signal band
TAUIMA  =               ! rtaui     ! Opacity image band
TATMIMA =               ! rtatmi    ! Atmosphere temp. image band
TREC    =               ! rtrec     ! Receiver temperature
MODE    =               ! rcmode    ! Calibration mode
FACTOR  =               ! ratfac    ! Applied calibration factor
ALTITUDE=               ! ralti     ! Site elevation
COUNTS(3)               ! rcount(3) ! Power of Atm., Chopp., Cold
KHEND
COMMENT * OBS.GAUSS : Gauss fit results (for spectra or drifts)
KHBEGIN = 'GAUSS'       ! (MXGAUS=5,mfit=3+6*MXGAUS)
NLINE   =               ! rnline    ! Number of components
SIGMA_BA=               ! rsigba    ! Sigma on base
SIGMA_RA=               ! rsigra    ! Sigma on line
FIT(3*NLINE)            ! rnfit(3*MXGAUS) ! Fit results
ERR(3*NLINE)            ! rnerr(3*MXGAUS) ! Errors
KHEND
KHEND   = 'CLASS'                       / End CLASS hierarchy
----------------------------------------------------------------------
(*1) Define a CLASS hierarchy
(*2) This structure is not required since the parameters can be derived
from a sequential reading of the following ones.
(*3) Here appears a rather general problem : the definition of arrays
into FITS. Common practice has been to use an index as in NAXISi for
example. But this is limited to small arrays in practice (size < 10
probably). In the list of original scan numbers (*4), the size may
be larger. This problem is fairly general and here is an opportunity
to address it. A possibility would be to use a single (non-indexed)
keyword. If the count has been previously defined, this solution is
unambigous: auto increment should simply be used by the reading
program. The two following examples would be valid, and correspond
to the following list of original scans added to produced the current
one : 100 to 110, 112 to 120, 345 to 360, 363 and 364, 400.
KHBEGIN ='HISTORY'
SEQUENCE= 5
START   = 100
REND    = 110
START   = 112
REND    = 120
START   = 345
REND    = 360
START   = 363
REND    = 364
START   = 400
REND    = 400
KHEND
KHBEGIN ='HISTORY'
SEQUENCE= 5
START   = 100
START   = 112
START   = 345
START   = 363
START   = 400
REND    = 110
REND    = 120
REND    = 360
REND    = 364
REND    = 400
KHEND
However this solution is very space consuming for long arrays.
Should table-like data be used for those array keywords? If so,
how could it be logically placed if hierarchies are implemented
as suggested by Bob Hanisch ? Could we use something more compact
such as
KHBEGIN ='HISTORY'
SEQUENCE= 5
START   = 100, 112, 345, 363, 400
REND    = 110, 120, 360, 364, 400
SEQUENCE= 1
START   = 500
REND    = 510
KHEND
This is non standard FITS in terms of alignment of formatted variables,
but can be easily read on any computer using a list directed I/O
Fortran format. The number of values in each line is defined by
keyword SEQUENCE here, so that the total number must be computed
by adding all values of the SEQUENCE keyword (note that the name
SEQUENCE is specific to the example. Other names would apply to other
cases).
(*5) XCOORD is used in CLASS for data (spectra or drifts) which are
not regularly sampled. It has been used for example to store in CLASS
format the IRAS LRS survey. The corresponding section contains
a unit for the array coordinates (integer code), followed by the
array of coordinates of the samples (here "coordinates" may be
frequency of course). This problem of irregularly sampled
data may be important in single-dish data as shown by the IRAS LRS
example. A logical solution to handle that in FITS format is the use
of standard extension tables.
In so far, we have not discussed the fate of multi-backend and/or
multi-beam observations.
For multi-backend, the use of AIPS-style FITS header seems to imply to
separate each backend data set in a different FITS file. An a priori
alternative would be to declare the frequency axis as randomly
sampled, but we then turn into several difficulties:
1) for reduced data, considering regularly sampled data as random ones
is obviously a lack of simplicity, and probably most processing systems
are not even able to do so (3_D VLM data cubes in AIPS with random
velociy axis?)
2) for raw data, it is likely that the calibration parameters depend
upon the backend. A specific structure would be required for
each backend.
A different FITS file for each backend is much simpler. The only disadvantage
is the loss of storage space, but the cost (and volume) of storage is
decreasing every day.
For multi-beam, the problem may be different, because the relative beam
positions are intrinsically random parameters, while the calibration
parameters are mostly identical for all beams. A suggestion would
be the use of GROUP data structure like in UVFITS. A few parameters
must be precised for each beam however (beam efficiency and beam sizes,
and possibly a scaling factor representing the average gain of the
backend part used for this beam).
Using GROUP data structure may also be possible for reduced (calibrated)
data, allowing to store a complete "random" map (regularly sampled maps
should use the normal axis coordinates). For raw data, this probably does
not make sense, as many calibration parameters will be valid only for
a few observations.
----------------------------------------------------------------------
In summary, following those guidelines, our proposal is (in priority order)
1) Always put the data in the FITS data area, NOT in an extension.
2) Put all important informations in a "standard" FITS header,
   based on the one used in AIPS for example. This "standard" header
   must be general enough to be understood by ALL FITS reading
   programs.
3) Adopt the Bob Hanisch proposal for hierarchy implementation and
   add a list of hierarchic keywords to describe completely the
   other informations. For simplicity reasons, this list may
   duplicate informations already present in the "standard" header.
4) Use table extensions for the coordinates of non regularly sampled data.
5) Use GROUP data structure for multi-beam observations, and possibly
   also for complete reduced "random" maps.
6) Find a solution for "array" keywords; we only suggested possible
   alternatives.
7) If possible, agree upon names for the hierarchical keywords,
   in order to ease data exchange between different single-dish
   reduction programs. (*) This does not preclude use of specific keywords
   for each analysis program, specially for data archive.
        Comments please...
(*) This is the lowest priority for us, because our FITS reading program
has the capability to define equivalent names for a keyword. This capability
is easy to implement in any program, and already saved us a lot of
additional programming time.

----------------------------------------------------------------------
Turning to the 6 questions asked by W.R. Burns in his original
letter, our answers are :
1) CLASS (Continuum and Line Analysis Simple System) for single-dish
   data,
   CLIC (Continuum and Line Interferometer Calibration) for IRAM
   interferometer data, with a data format which is an extension
   of the CLASS format
   GILDAS (Grenoble Image and Line Data Analysis Software) for
   image processing.
   and AIPS as a reference in interferometry.
2) CLASS CLIC and GILDAS are developped in collaboration between
   IRAM and GAG, and the software is now used on many sites.
3) External support, limited to bug repairs by lack of manpower.
   Enhancement done in-house on a "urgent need" basis.
   In house software works only under VMS up to now.
4) Hardware : fast workstations, probably UNIX
   Software : UNIX conversion, may be X-windows, upgrade of
   image processing to handle more specifically IRAM Plateau
   de Bure data, and more homogenously 30-m and Plateau de Bure
   data.
5) We use AIPS, and distribute our software. Sharing code with
   some standardization would seem unpractical given our limited
   manpower (however if anybody is interested by our small code
   to get equivalent names, it can be done of course!).
6) Of course, single dish data could make use of other packages. This
   is precisely the goal of our proposal points 1 and 2.
PS: Stephane Guilloteau is unable to attend the meeting, and Thierry
Forveille will be the IRAM representative.

----------------------------------------------------------------------
Appendix : data structure (in VAX-VMS FORTRAN syntax) used for
single-dish data in CLASS format, with corresponding standard
FORTRAN variable names and physical meaning:
*
        STRUCTURE /OBSERVATION_DESCR/
*
* OBS.POINTER : Length of sections. If 0, section not present.
            STRUCTURE /POINTER_DESCR/ POINTER
                INTEGER GENERAL
                INTEGER POSITION
                INTEGER SPECTRO
                INTEGER FSWITCH
                INTEGER DRIFT
                INTEGER BEAM
                INTEGER CALIBRATION
                INTEGER SKYDIP
                INTEGER GAUSS
                INTEGER HFS
                INTEGER SHELL
                INTEGER CONTINUUM
                INTEGER HISTORY
                INTEGER PLOT
                INTEGER BASE
                INTEGER XCOORD
            END STRUCTURE
*
* Section -2    /CRPAR/
* OBS.GENERAL : General parameters, always present
            STRUCTURE /GENERAL_DESCR/ GENERAL   ! (mgen=9,mgen2=11)
                INTEGER NUMBER                  ! rnum      ! Observation number
                INTEGER VERSION                 ! rver      ! Version number
                CHARACTER*12 TELESCOPE          ! rteles(3) ! Telescope name
                INTEGER DATE_OBS                ! rdobs     ! Date of observatio
n
                INTEGER DATE_RED                ! rdred     ! Date of reduction
                INTEGER SYSTEM                  ! rtypec    ! Type of coordinate
s
                INTEGER KIND                    ! rkind     ! Type of data
                INTEGER QUALITY                 ! rqual     ! Quality of data
                INTEGER SCAN                    ! rscan     ! Scan number
                REAL*8  UT_TIME                 ! rut       ! UT of observation
                REAL*8  LST_TIME                ! rst       ! LST of observation
                REAL*4  AZIMUTH                 ! raz       ! Azimuth
                REAL*4  ELEVATION               ! rel       ! Elevation
                REAL*4  TAU                     ! rtau      ! Opacity
                REAL*4  TSYS                    ! rtsys     ! System temperature
                REAL*4  INTEGRATION             ! rtime     ! Integration time
            END STRUCTURE
*
* Section -3
* OBS.POSITION : Position information.
            STRUCTURE /POSITION_DESCR/ POSITION ! (mpos=11)
                CHARACTER*12 SOURCE             ! rsourc(3) ! Source name
                REAL*4  EPOCH                   ! repoch    ! Epoch of coordinat
es
                REAL*8  LAMBDA                  ! rlam      ! Lambda
                REAL*8  BETA                    ! rbet      ! Beta
                REAL*4  LAMBDA_OFF              ! rlamof    ! Offset in Lambda
                REAL*4  BETA_OFF                ! rbetof    ! Offset in Beta
                INTEGER PROJECTION              ! rproj     ! Projection system
            END STRUCTURE
*
* Section -4
* OBS.SPECTRO : Spectroscopic information (for spectra)
            STRUCTURE /SPECTRO_DESCR/ SPECTRO   ! (mspec=15)
                CHARACTER*12 LINE               ! rline(3)  ! Line name
                REAL*8  REST                    ! rrestf    ! Rest frequency
                INTEGER NCHAN                   ! rnchan    ! Number of channels
                REAL*4  REFERENCE               ! rrchan    ! Reference channels
                REAL*4  FRES                    ! rfres     ! Frequency resoluti
on
                REAL*4  FOFF                    ! rfoff     ! Frequency offset
                REAL*4  VRES                    ! rvres     ! Velocity resolutio
n
                REAL*4  VOFF                    ! rvoff     ! Velocity at refere
nce channel
                REAL*4  BAD                     ! rbad      ! Blanking value
                REAL*8  IMAGE                   ! rimage    ! Image frequency
                INTEGER VELOCITY                ! rvtype    ! Type of velocity
            END STRUCTURE
*
* Section -5
* OBS.BASE : Baseline information (for spectra of drifts)
            STRUCTURE /BASE_DESCR/ BASE         ! (mwind=20,mbase=4+2*mwind)
                INTEGER DEGREE                  ! rdeg      ! Degree of last bas
eline
                REAL*4  SIGMA                   ! rsigfi    ! Sigma
                REAL*4  AREA                    ! raire     ! Area under windows
                INTEGER NWINDOW                 ! rnwind    ! Number of line win
dows
                REAL*4  WIND1(MWIND)            ! rw1(mwind)! Lower limits of wi
ndows
                REAL*4  WIND2(MWIND)            ! rw2(mwind)! Upper limits of wi
ndows
            END STRUCTURE
*
* Section -6
* OBS.HISTORY : Scan numbers of initial observations
            STRUCTURE /HISTORY_DESCR/ HISTORY   ! (mseq=100,morig=2*mseq+1)
                INTEGER SEQUENCE                ! rnseq     ! Number of sequence
s
                INTEGER START(MSEQ)             ! rstart(mseq) ! Start can numbe
r of seq.
                INTEGER END(MSEQ)               ! rend(mseq)   ! End scan number
 of seq.
            END STRUCTURE
*
* Section -7
* OBS.PLOT : Default plotting limits.
            STRUCTURE /PLOT_DESCR/ PLOT         ! (mplot=4)
                REAL*4  YMIN                    ! ramin     ! Min Y value plotte
d
                REAL*4  YMAX                    ! ramax     ! Max Y value plotte
d
                REAL*4  XMIN                    ! rvmin     ! Min X value plotte
d
                REAL*4  XMAX                    ! rvmax     ! Max X value plotte
d
            END STRUCTURE
*
* Section -8
* OBS.FSWITCH : Frequency switching information (for spectra)
            STRUCTURE /FSWITCH_DESCR/ FSWITCH   ! (mxphas=5,mfsw=1+4*mxphas)
                INTEGER NPHASE                  ! rnphas    ! Number of phases
                REAL*8  DECALAGE(MXPHAS)        ! rdecal(mxphas) ! Frequency off
sets
                REAL*4  DUREE(MXPHAS)           ! rduree(mxphas) ! Time per phas
e
                REAL*4  POIDS(MXPHAS)           ! rpoids(mxphas) ! Weight of eac
h phase
            END STRUCTURE
*
* Section -14
* OBS.CALIBRATION : Calibration parameters
            STRUCTURE /CALIBRATION_DESCR/ CALIBRATION   ! (mcalib=19)
                REAL*4  BEAM_EFF                ! rbeeff    ! Beam efficiency
                REAL*4  FORW_EFF                ! rfoeff    ! Forward efficiency
                REAL*4  GAIN_IM                 ! rgaini    ! Image/Signal gain
ratio
                REAL*4  H2OMM                   ! rh2omm    ! MM of water vapor
                REAL*4  PAMB                    ! rpamb     ! Ambient pressure (
hPa)
                REAL*4  TAMB                    ! rtamb     ! Ambient temperatur
e (K)
                REAL*4  TATMSIG                 ! rtatms    ! Atmosphere temp. s
ignal band
                REAL*4  TCHOP                   ! rtchop    ! Chopper temperatur
e
                REAL*4  TCOLD                   ! rtcold    ! Cold load temperat
ure
                REAL*4  TAUSIG                  ! rtaus     ! Opacity signal ban
d
                REAL*4  TAUIMA                  ! rtaui     ! Opacity image band
                REAL*4  TATMIMA                 ! rtatmi    ! Atmosphere temp. i
mage band
                REAL*4  TREC                    ! rtrec     ! Receiver temperatu
re
                INTEGER MODE                    ! rcmode    ! Calibration mode
                REAL*4  FACTOR                  ! ratfac    ! Applied calibratio
n factor
                REAL*4  ALTITUDE                ! ralti     ! Site elevation
                REAL*4  COUNTS(3)               ! rcount(3) ! Power of Atm., Cho
pp., Cold
            END STRUCTURE
*
* Section -16
* OBS.SKYDIP : For Skydips observations. No associated data.
            STRUCTURE /SKYDIP_DESCR/ SKYDIP     ! (msky=10,mskydip=10+4*msky)
                CHARACTER*12 LINE               ! rsline(3) ! Line name
                REAL*8  REST                    ! rsrest    ! Rest frequency
                REAL*8  IMAGE                   ! rsimag    ! Image frequency
                INTEGER NSKY                    ! rnsky     ! Number of points o
n sky
                INTEGER NCHOP                   ! rnchop    !    -   -    -    -
 chopper
                INTEGER NCOLD                   ! rncold    !    -   -    -    -
 cold load
                REAL*4  ELEVATION(MSKY)         ! relev(msky)  ! Elevations
                REAL*4  EMISSION (MSKY)         ! remiss(msky) ! Power on sky
                REAL*4  CHOPPER  (MSKY)         ! rchopp(msky) ! Power on choppe
r
                REAL*4  COLD    (MSKY)          ! rcold(msky)  ! Power on cold l
oad
            END STRUCTURE
*
* Section -9    /CRGAUS/
* OBS.GAUSS : Gauss fit results (for spectra or drifts)
            STRUCTURE /GAUSS_DESCR/ GAUSS       ! (MXGAUS=5,mfit=3+6*MXGAUS)
                INTEGER NLINE                   ! rnline    ! Number of componen
ts
                REAL*4  SIGMA_BASE              ! rsigba    ! Sigma on base
                REAL*4  SIGMA_RAIE              ! rsigra    ! Sigma on line
                REAL*4  FIT(3*MXGAUS)           ! rnfit(3*MXGAUS) ! Fit results
                REAL*4  ERR(3*MXGAUS)           ! rnerr(3*MXGAUS) ! Errors
            END STRUCTURE
*
* Section -12   /CRSHEL/
* OBS.SHELL : "Stellar shell" profile fit results (for spectra)
            STRUCTURE /SHELL_DESCR/ SHELL       ! (mshell=43)
                INTEGER NLINE                   ! rlshel    ! Number of componen
ts
                REAL*4  SIGMA_BASE              ! rbshel    ! Sigma on base
                REAL*4  SIGMA_RAIE              ! rrshel    ! Sigma on line
                REAL*4  FIT(20)                 ! rnshel(20)! Fit results
                REAL*4  ERR(20)                 ! reshel(20)! Errors
            END STRUCTURE
*
* Section -13   /CRNH3/
* OBS.HFS : "Hyperfine Structure" profile fit results (for spectra)
*           (e.g. NH3, HCN)
            STRUCTURE /HFS_DESCR/ HFS           ! (mnh3=27)
                INTEGER NLINE                   ! rlnh3     ! Number of componen
ts
                REAL*4  SIGMA_BASE              ! rbnh3     ! Sigma on base
                REAL*4  SIGMA_RAIE              ! rrnh3     ! Sigma on line
                REAL*4  FIT(12)                 ! rnnh3(12) ! Fit results
                REAL*4  ERR(12)                 ! renh3(12) ! Errors
            END STRUCTURE
*
* Section -10   /CRCONT/
* OBS.DRIFT : Continuum drift description (for drifts)
            STRUCTURE /DRIFT_DESCR/ DRIFT       ! (mcont=16)
                REAL*8  REST                    ! rfreq     ! Rest frequency
                REAL*4  WIDTH                   ! rwidth    ! Bandwidth
                INTEGER NPOINT                  ! rnpoin    ! Number of data poi
nts
                REAL*4  REFERENCE               ! rrpoin    ! Reference point
                REAL*4  TIMOFF                  ! rtref     ! Time at reference
                REAL*4  ANGOFF                  ! raref     ! Angular offset at
ref.
                REAL*4  PA                      ! rapos     ! Position angle of
drift
                REAL*4  TIMRES                  ! rtres     ! Time resolution
                REAL*4  ANGRES                  ! rares     ! Angular resolution
                REAL*4  BAD                     ! rcbad     ! Blanking value
                INTEGER SYSTEM                  ! rctype    ! Type of offsets
                REAL*8  IMAGE                   ! rcimag    ! Image frequency
                REAL*4  COLLA                   ! rcolla    ! Collimation error
Az
                REAL*4  COLLE                   ! rcolle    ! Collimation error
El
            END STRUCTURE
*
* Section -11
* OBS.BEAM : Beam-switching parameters (for spectra or drifts)
            STRUCTURE /BEAM_DESCR/ BEAM         ! (mbeam=5)
                REAL*4  AZIMUTH                 ! rcazim    ! Azimuth of observa
tion
                REAL*4  ELEVATION               ! rcelev    ! Elevation of obser
vation
                REAL*4  THROW                   ! rspace    ! Beam spacing
                REAL*4  PA                      ! rbpos     ! Position angle of
beams
                INTEGER SYSTEM                  ! rbtype    ! System for angle
            END STRUCTURE
*
* Section -15 /CRPOINT/
* OBS.CONTINUUM : Double gaussian and baseline fit results (for drifts)
            STRUCTURE /CONT_DESCR/ CONTINUUM    ! (mfcont=19)
                INTEGER NLINE                   ! rlcont    ! Number of componen
ts
                REAL*4  SIGMA_BASE              ! rbcont    ! Sigma on base
                REAL*4  SIGMA_RAIE              ! rrcont    ! Sigma on line
                REAL*4  FIT(8)                  ! rncont(8) ! Results
                REAL*4  ERR(8)                  ! recont(8) ! Errors
            END STRUCTURE
        END STRUCTURE
*
From dwells@fits.CX.NRAO.EDU Mon Oct 16 10:03:50 1989
Received: from NRAO.EDU (cv3.cv.nrao.EDU) by fits.cx.nrao.edu (4.0/SMI-DDN)
	id AA05546; Mon, 16 Oct 89 10:03:48 EDT
Received: from fits.cx.nrao.edu by NRAO.EDU (3.2/DCW-2n    )
	id AA09306; Mon, 16 Oct 89 10:03:40 EDT
Received: by fits.cx.nrao.edu (4.0/SMI-DDN)
	id AA05491; Mon, 16 Oct 89 09:55:57 EDT
Return-Path: <dwells>
Message-Id: <8910161355.AA05485@fits.cx.nrao.edu>
Status: RO
From: dwells@fits.CX.NRAO.EDU (Don Wells)
Sender: dishfits-request@fits.CX.NRAO.EDU
To: dishfits@fits.CX.NRAO.EDU
Cc: dwells@fits.CX.NRAO.EDU
Subject: Re:  Single-dish FITS
Date: Mon, 16 Oct 89 09:55:54 EDT
Dear friends,
The Guilloteau and Forveille paper is a perceptive and sophisticated
discussion of FITS issues for single-dish astronomy. It is a valuable
contribution in preparation for the Green Bank workshop.
Here are some comments, corrections, differences of opinion:
1. re matrices versus tables:
Stephane and Thierry stress that data transfer from single-dish
systems to "general" IP systems should be by means of images (n-dimensional
matrices) in classical FITS form. I agree that *if* the data are an
n-dimensional matrix then the Basic FITS format should be used. However,
I am optimistic that we will soon have effective interchange capability
for other types of data structures, particularly tables. I contend that
we must make the effort to build and demonstrate this capability in many
systems so that astronomical data interchange and archiving will no longer
be confined to matrices. I would like to see the single-dish community take
a leadership role in this effort.
2. re list-directed-READ for FITS value fields:
In March 1979 Eric Greisen and I agreed that FITS keyword value fields
would use the notation of FORTRAN list-directed READs. The list-directed
READ design allows for lists of values, and Eric and I understood what we
were proposing. My memory is that when the design went to the review panel
in April-May 1979 there were strong protests that many computer systems
were not sophisticated enough to cope with completely free-field headers,
that our design was too sophisticated and that it ran the risk of being
ignored by many groups in astronomy. We compromised. We stopped talking
about multi-valued keywords and we even gave up the free-field concept ---
we specified that the required keywords of a FITS header must be in a rigid
format so that unsophisticated readers would have an easier job.
The multiple-value problem is fundamental. I do not have a nice, neat
answer for it. In headers we have another problem: Eric and I decided
that all standard header lines would fit into one "card". This limits how
many values a keyword can have. I believe now that the best solutions will
use tables, with separate tuples for the values.
3. re irregularly sampled data:
The classical answer to this problem is the random-groups format.
Indeed, since 1980 the leading FITS designers have all been somewhat sad
that we did not think of random-groups until after the Basic FITS Agreement
had already been implemented in production systems. We wish that Basic FITS
had been the random-groups format, so that simple matricies would be a
special case. In 1984 we made the Generalized Extensions Agreement have this
form.
Jim Condon of NRAO and his collaborators have made several surveys of the
northern sky with the 300-foot in a rapid scanning mode using a seven-beam
feed. The randomly sampled data were formatted as random-groups FITS.
AIPS is able to read such data (see relevant sections of Going AIPS for details)
and there are tasks which can grid the samples into maps of the sky. Indeed,
the tasks are variations on the aperture synthesis task UVMAP (the Fourier
transform step is skipped). The random samples are sorted using task UVSRT.
I.e., FROM THE POINT OF VIEW OF A PROGRAMMER, RANDOM SAMPLES OF FLUX ON THE
SKY ARE STRUCTURALLY ALMOST IDENTICAL TO THE RANDOM SAMPLES OF THE UV PLANE
ACQUIRED IN APERTURE SYNTHESIS, AND SIMILAR SOFTWARE APPROACHES ARE FEASIBLE.
Note that scanning infrared satellite data (IRAS) and high-energy photon
event data fall in this same class. Although the 300-foot surveys were
continuum data only I believe that the AIPS tasks are more general, that
they can sort and grid spectral-line surveys.
Bill Cotton's "3-D Binary Tables" design, which he devised for
VLBA calibration, is yet another approach to the irregularly sampled data
problem. It is not as elegant mathematically, but the bit stream that it
produces is essentially *identical* to that produced by random groups (so
that there is no difference in efficiency) and he can transmit data types
that random-groups cannot handle.
4. re hierarchical keyword syntax:
Bob Hanisch has raised two issues: (1) "Do we really need a hierarchical
namespace?" and (2) "What syntax should we use?". I still believe that
the FITS community should endeavour to transmit *MEANING* (semantics) as well
as mere *DATA* (syntax), and I believe that it will be necessary to organize
our name-space in order to achieve this goal reliably. However I am grateful
to Bob for having the courage to go against the common wisdom in this area,
to make us think about the issue. Regarding syntax, there are two proposals.
Either we devise rules for what I call the "push-down-stack" syntax (the
concept discussed by Hanisch and endorsed by Guilloteau and Forveille), or
we adapt the existing "domain-name-system" rules for single keyword=value
header cards. I think that either approach can probably be made towork in
the end, but I have more confidence in the second approach.
I have a paper of my own which has been in preparation for several weeks,
and I expect to broadcast it soon.
Regards,
Don
Donald C. Wells, Associate Scientist | NSFnet: dwells@nrao.edu [192.33.115.2]
National Radio Astronomy Observatory | SPAN:   NRAO::DWELLS    [6654::]
Edgemont Road                        | BITnet: DWELLS@NRAO
Charlottesville, VA 22903-2475 USA   | UUCP:   ...!uunet!nrao.edu!dwells
+1-804-296-0277  (38:02.2N/78:31.1W) | TWX=510-587-5482, Fax=+1-804-296-0278
From dwells@fits.CX.NRAO.EDU Mon Oct 16 17:23:36 1989
Received: from NRAO.EDU (cv3.cv.nrao.EDU) by fits.cx.nrao.edu (4.0/SMI-DDN)
	id AA06260; Mon, 16 Oct 89 17:23:28 EDT
Received: from fits.cx.nrao.edu by NRAO.EDU (3.2/DCW-2n    )
	id AA10347; Mon, 16 Oct 89 17:22:44 EDT
Received: by fits.cx.nrao.edu (4.0/SMI-DDN)
	id AA06207; Mon, 16 Oct 89 17:03:31 EDT
Return-Path: <dwells>
Message-Id: <8910162103.AA06200@fits.cx.nrao.edu>
Status: RO
From: dwells@fits.CX.NRAO.EDU (Don Wells)
Sender: dishfits-request@fits.CX.NRAO.EDU
To: dishfits@fits.CX.NRAO.EDU
Subject: Binary Table Structure for Spectra
Date: Mon, 16 Oct 89 17:03:26 EDT
Dear Friends-of-FITS:
The paper below is a proposal to the single-dish radio community,
which could adopt it even if the rest of astronomy doesn't want to do so.
I, of course, hope that it may be possible to forge an alliance across
all of astronomical spectroscopy.
My actual proposal is only the first 9 pages. The remaining 23 pages are
appendices containing the text of the Floating Point Agreement, the
Grosbol/Wells Hierarchical Keyword draft proposal and a detailed
description of AIPS support for single-dish random-scan data and
Cotton's 3-D Binary Table extension format.
Donald C. Wells, Associate Scientist | NSFnet: dwells@nrao.edu [192.33.115.2]
National Radio Astronomy Observatory | SPAN:   NRAO::DWELLS    [6654::]
Edgemont Road                        | BITnet: DWELLS@NRAO
Charlottesville, VA 22903-2475 USA   | UUCP:   ...!uunet!nrao.edu!dwells
+1-804-296-0277  (38:02.2N/78:31.1W) | TWX=510-587-5482, Fax=+1-804-296-0278
------------------- LaTeX, 32 pages, cut here -------------------------
\documentstyle[11pt,twoside]{article}
\pagestyle{headings}
\raggedbottom
%
\begin{document}
\title{A Binary Table Structure for Spectra}
\author{Donald C. Wells\\
{\small \raisebox{0.1cm}{\rule{0cm}{0.3cm}}NRAO, Charlottesville, VA}\\
{\small 804-296-0277, {\tt dwells@nrao.edu}}}
\date{{\em DRAFT} \today}
\maketitle
\begin{abstract}
The 3-D binary table FITS extension is proposed as an efficient format
to transmit groups of astronomical spectra and associated parameters.
Syntactic conventions for keywords and table column labels are
suggested.
\end{abstract}
\tableofcontents
\clearpage
\section{The Problem}
The basic FITS (Flexible Image Transport System) Agreement of 1979
defined a syntax for transmitting N-dimensional matricies. It was (and
still is) perfectly capable of transmitting spectra, for which N=1, as
vectors of binary data described by ASCII headers using the
keyword=value notation. However, such usage is inefficient: in most
cases more bytes are needed for the header than for the binary vector!
Spectroscopy observations generally need more ancillary parameters,
more keyword=value pairs, than do imaging observations and this only
exaggerates the disproportion between header and data for
spectroscopy. The basic FITS format is simply not efficient for
spectroscopy applications.
Spectroscopy observations often involve large numbers of short
integrations, which implies that large numbers of ASCII headers must
be written, transmitted, archived and processed. Although many people
have wished that the vectors as well as the headers could be
transmitted in ASCII form so that {\em everything} would be
human-readable, this is simply not a practical option for our new data
systems in which more and more feeds are analyzed by spectrometers
with higher and higher resolution.  Binary notation is essential for
the spectra, and we must strive to reduce the overhead of the headers.
This means that even the headers must be in binary, and redundancy
must be minimized.
\section{The Solution}
We need binary data structures which include the ancillary parameters
in binary form, bound to the binary vectors.  These structures must
be described in FITS style in order to make them {\em flexible.} The
ASCII description will require almost as many bytes as would a basic
FITS header. To minimize this overhead we need to be able to transmit
{\em arrays} of the structures. In this concept only one copy of the
ASCII description is needed for an arbitrarily large number of binary
spectra plus parameters, all grouped together in one FITS file.
There are two existing FITS formats that have the desired property:
the random groups format\footnote{Greisen, E.W., Harten, R.H.: 1981,
Astron. Astrophys. Suppl. Ser. 44, 371, "An Extension of FITS for
Groups of Small Arrays of Data".} and the 3-D binary tables
format\footnote{Cotton, circa 1986, coded in AIPS; see
Appendix~\ref{app:C3DBTF} and Section~\ref{sect:C3DBTF}.}. There is no
question that random groups are a feasible solution, but binary 3-D
tables are preferable, for two reasons. First, experience shows that
many people have difficulty understanding the random groups concept
but everyone claims that they understand tables intuitively. Second,
the random groups format requires that the same data type be used for
both spectra and parameters but the 3-D binary tables allow mixed data
types, a great convenience and valuable notational power. I propose
that Cotton's 3-D binary table format be adopted for astronomical
spectroscopy data interchange.
\section{Conventions}
The basic concept that I am proposing is that a set of spectroscopic
observations be regarded as a {\em table}, with the spectral vectors
appearing as one of the columns of the table and the ancillary
parameters appearing in other columns. The following subsections
propose certain conventions for the organization of the table.
\subsection{Minimizing Redundancy}\label{sect:redundant}
Many of the parameters associated with a set of spectroscopic
observations are constant for the whole set. If each such parameter
must appear in a  separate column of the table the same bytes will
appear over and over. I propose that such parameters be moved to the
extension header and encoded as keyword=value, and that they be
regarded as a {\em virtual column.}
\subsection{Multiple Feeds}
Many spectroscopic instruments today observe several positions on the
sky simultaneously. There are two possible encoding conventions.
First, we could encode such observations as one structure containing
several binary vectors, with many parameters shared between the
spectra and many other parameters appearing once for each of the
spectra. Alternatively, we could encode the several spectra as
separate structures, with the parameters which are identical being
duplicated but without the notational complexity of the multiple
instances of spectra and parameters in one structure. I believe that
the latter case, the separate structure scheme, is easier to design
and to implement, and so I propose that it be adopted.
Some spectrometers measure more than one band simultaneously. This
case is analogous to multiple feeds and the separate structure
approach handles it with no new notation.
In some cases datasets contain spectra with different numbers of
channels. I propose that the number of channels be specified in a
column. If all spectra have the same length the column need not be
carried with the spectra in the structures --- it can be given once in
the extension header (see section \ref{sect:redundant}).
\subsection{Hierarchical Table Labels}
Table column labels pose the same problems of potential name conflict
as do keywords.  It appears that the only feasible solution will be
some kind of hierarchical classification scheme. The two draft
proposals available currently are the Hierarchical Keyword Agreement
(the ``domain-name-system'' proposal) of Grosb{\o}l and Wells, whose
text is given below in Appendix~\ref{app:HKA}, and the
``push-down-stack'' proposal of Tody and Hanisch, as discussed in
Hanisch's DISHFITS-exploder message of 05~October. In the remainder of
this paper I will discuss the Grosb{\o}l/Wells notation because it
leads to a natural generalization for table labels. I have not yet
decided whether the Tody/Hanisch proposal offers an equally suitable
solution for this application.
The FITS ASCII Tables standard\footnote{Harten, R.H., Grosb{\o}l, P.,
Greisen, E.W., Wells, D.C.: 1988, Astron.  Astrophys. Suppl. Ser. 73,
365, "The FITS Tables Extension".} does not specify any limit
on the length of column label strings. This is yet another example of
the continuing need for interpretation of the rules of the standards.
Let us assume that the string-value rule of the Basic FITS
Agreement\footnote{Wells, D.C., Greisen, E.W., Harten, R.H.: 1981,
Astron. Astrophys. Suppl.  Ser. 44, 363, "FITS: A Flexible Image
Transport System".  See p.~364-365.} applies: string values may be of
any length which fits on one card (80 characters), but should be
unique/useful if truncated to 8 characters.  I therefore propose that
hierarchical classification codes be used to obtain uniqueness in the
labels, {\em but that the order be reversed so that the last
substring, the keyword, will be in the first 8 characters.} If the
same code string is used as a keyword in the extension header, as
proposed in Section~\ref{sect:redundant} above, it will appear in
normal order, with the {\tt HIERARCH} code in the first 8 columns and
the keyword at the end of the string, just before the equals sign.
\subsection{Units}
I strongly recommend that only SI units be used in data interchange in
astronomy, as specified in the Basic FITS Agreement.
\subsection{Spectral Format Table Labels}
I propose that column names and parameter values should be the keyword
and axis-type names used previously in FITS designs, wherever
possible. By this I mean that in situations where in other proposals
keywords are used in FITS headers that those same names be used in
table column labels for tables of spectra, and that where various
conventions are specified for the value fields of those keywords that
those same conventions be used for the value fields in those columns.
Thus, I am proposing that the parameter columns be regarded as {\em
virtual header cards}.
\section{Status of Relevant Draft Standards}
\subsection{Generalized Extensions, ASCII Tables}
The two FITS standards were published in Astron.~Astrophys.~Suppl. in
1988, and were approved by IAU Commission~5 at the Baltimore meeting
in August 1988.\footnote{IAU Resolution B1, p.~10,
IAU Information Bulletin 61, January 1989.}
\subsection{The Floating Point Agreement} The FPA has been discussed
for at least four years in committees. The prototype implementations
were done circa 1985-6. The final text was adopted 28~February~1989,
and is given in Appendix~\ref{app:FPA}. The FPA was endorsed by the
European Working Group for Coordination of Astronomical Software in
May 1989, and by the AAS Working Group for Astronomical Software in
June 1989. The Agreement is now under consideration by the IAU FITS
Working Group, which is awaiting the final interchange and
interoperability proof for the 64-bit data type.  Activation is
anticipated for 1990.
\subsection{Binary 3-D Table Format}\label{sect:C3DBTF}
Bill Cotton designed his table format as an extension of the standard
ASCII table format in order to represent certain complicated
calibration table structures needed for the VLBA analysis software.
The only description currently available is from section 14.5.3 of
``Going AIPS'', the AIPS programmer's reference manual. This text has
been extracted from the AIPS directories and is given below in
Appendix~\ref{app:C3DBTF}.  This extension was implemented in AIPS
circa 1986, and code supporting it has been operational on several
hundred computers for several years now. The extension was implemented
in MIDAS circa 1987, and AIPS-MIDAS interoperability trials have been
made. As of 1989 the extension is being implemented as a format for
photon lists in high energy astronomy, first for the ROSAT data
analysis software.  The 3-D Binary Table design is generally believed
by the FITS standards committees to be the draft proposal ``most
likely to succeed''. Only one or two trivial changes are anticipated
when it finally passes the IAU sometime in the future.
\subsection{Hierarchical Keyword Agreement} The original idea was
implemented by Eric Greisen in the first FITS software for the
NRAO-Charlottesville IBM synthesis mapping package in 1979. A more
recent prototype circa 1986-7 is the SNGLDSH format at NRAO. The
concept has been discussed in the FITS committees since 1987. A
current draft proposal by Grosb{\o}l and Wells dates from
28~February~1989; it is reproduced below in Appendix~\ref{app:HKA}.
Alternative concepts are still being offered. At present no clear
consensus exists favoring any one proposal.
\section{An Example Showing Packing Efficiency}
The preceding discussion has proposed Cotton's FITS extension as a
solution appropriate for all spectroscopic applications in astronomy.
At present the main astronomy discipline considering this problem is
single-dish radio astronomy. There has been much preliminary work
involving keywords, syntax and even semantics for this application,
and it need not be repeated here. Indeed, even the header syntax for
the 3-D extension is well documented in an appendix to this paper, and
so I choose not to give a full example.  The important point to be
addressed is the {\em packing efficiency} which is possible for the
proposed design..
Consider a radio spectrometer with 1024 channels in floating point
form. A data file will start with a basic FITS header which has no
data matrix ({\tt NAXIS=0}). This will be followed by the extension
header, which will specify {\tt XTENSION='A3DTABLE'} (once the IAU
approves this extension the value will probably be {\tt '3DTABLE'}).
Each spectrum will be encoded as a binary data structure which is
described with the notational machinery of the 3-D tables. A grossly
simplified data structure which has an example of each data type might
be:
\begin{center}
\begin{tabular}{|l|l|l|l|l|l|l|}
\hline
label:	&{\tt SCAN}	&{\tt SOURCE}	&{\tt BASEFREQ}
&{\tt DELTFREQ}	&{\tt NCHAN}	&{\tt SPECTRUM}	\\
\hline
format:	&{\tt 1J}	&{\tt 20A}	&{\tt 1D}
&{\tt 1F}	&{\tt 1I}	&{\tt 1024F} \\
\hline
bytes:	&4		& 20		& 8
& 4		& 2		& 4096 \\
\hline
\end{tabular}
\end{center}
The {\tt SCAN} number is a 32-bit integer, the {\tt SOURCE} name is a
string, the frequency of the reference channel (in Hertz, of course)
is a double precision number and the frequency increment is single
precision. The spectrum itself is single precision floating point and
the number of channels {\tt NCHAN} is specified by a 16-bit integer
(note that the {\em maximum} number of channels is given by the format
of the {\tt SPECTRUM} field; {\tt NCHAN} might be less than this
number). {\em The column labels used here are not to be regarded as a
part of my proposal; rather, if my 3-D table concept is accepted for
astronomical spectra an ad hoc task force should negotiate these
details.} The binary data types can be flexible: one site might encode
{\tt SCAN} numbers as 16-bit integers while another used 32-bits.
Likewise, some implementations may choose to encode spectra using
16-bit integers rather than 32-bit reals in order to save bytes in the
files (the FITS machinery for integer scaling permits this to work
even for inherently non-integer values).
For this example there will be 2880 bytes in the basic header, 5760
bytes in the extension header, 4134 bytes per spectrum and one
tapemark.  It is true that a real spectral observation will probably
need about 50 parameters, not 5, but even at 4 bytes per parameter
this will add only 200 more bytes per spectrum. Therefore, it will
still be true that for one spectrum the header overhead will dominate.
Indeed, 50 more columns in the table will require several more logical
records of extension header to document, making the imbalance even
worse!  However, additional spectra will add no more overhead. If the
table contains as many as 20 spectra the efficiency will exceed 90\%.
\section{Unresolved Issues, Other Ideas}
\begin{itemize}
\item{\bf Hierarchical Keywords:} There is no consensus yet on the
syntax, or even on the necessity of a hierarchical data space.
\item{\bf Vocabulary, Semantics, Units:} This proposal is only
concerned with the syntax for transmitting spectra. Each discipline of
astronomy must agree on the vocabulary and on the semantics and units
for its special language.
\item{\bf World Coordinates:} We must agree on conventions for
describing the mapping between pixel number and frequency in the
spectra. I recommend that we utilize notations similar to those
devised by Eric Greisen for imagery. This means that we need to
identify how many types of mappings occur in astronomical instruments
and assign names to them.  We must analyze their mathematical forms to
determine how many parameters each has and attempt to devise a uniform
convention for the notation.  An important question will be whether
all of astronomy can agree to use frequency as the independent
variable for spectroscopic data interchange; if not, we will have both
wavelength and frequency (wavenumbers are just frequency by another
name, of course). All of these world coordinates issues are beyond the
scope of this paper.
\item{\bf Higher-Level Organization of Datasets:} Multiple named table
extensions may be grouped together in one file --- this provides a
higher level organizational tool. The Generalized Extensions standard
also specifies keywords and conventions for a hierarchical structure
of extensions, which is yet another higher level organizational tool.
\item{\bf Auxilliary Tables:} The FITS table extensions could be
used to transmit calibrations, source lists, etc., in addition to the
spectra --- this could permit the elimination of even more redundancy
in the sense of ``normalizing'' a relational DBMS schema.
\end{itemize}
\appendix
% ==================================================================
\clearpage
\section{Floating Point Proposal for FITS}\label{app:FPA}
\begin{center}Don Wells, Preben Grosb{\o}l\end{center}
\begin{center}28 February 1989\footnote{This
text is very slightly revised from the version that was endorsed by
the European FITS Committee in April 1989 and by the AAS Working Group
for Astronomical Software in June 1989. It will now be reviewed by the
IAU FITS Working Group for adoption as a part of the FITS standards,
and will probably be activated in 1990.}\end{center}
\subsection{The Proposal}
The Basic FITS, Random Groups and Generalized Extensions Agreements
are revised to add IEEE-754 32- and 64-bit floating point numbers to
the original set of FITS data types.  BITPIX=-32 and BITPIX=-64
signify 32- and 64-bit IEEE floating point numbers; the absolute value
of BITPIX is used for computing the sizes of data structures. The full
IEEE set of number forms are allowed for FITS interchange, including
all special values (e.g., the 'not-a-number' cases). The order of the
bytes will be sign and exponent first, followed by the mantissa bytes
in order of decreasing significance.  The BLANK keyword will be
ignored by FITS readers when BITPIX=-32 or -64.
\subsection{Discussion}
The dynamic ranges of astronomical data have increased steadily in the
years since the original FITS Agreement (March 1979) which defined 8-, 16-
and 32-bit integers as the only data formats. All integer formats are
limited by an absolute error, whereas floating point formats can span a
much wider numerical range with only a relative error. Floating point
formats are desirable for the applications that need increased dynamic
range. Many astronomical data systems now use floating point in data
processing, and conversion to and from FITS integer formats sacrifice
accuracy and dynamic range and consume significant computer time.  Almost
all of the new computers designed since about 1981 use the IEEE format
(see references below). We are proposing that FITS be revised to allow
transmission of 32- and 64-bit floating point data within the FITS format
using the IEEE standard. This Floating Point Agreement will also apply to
the Random Groups Extension and to those Generalized Extensions for which
BITPIX is not explicitly restricted (e.g., BITPIX=8 for XTENSION='TABLE').
In order to transmit floating point data in the main data matrix we must
add some new convention to the main FITS header. We propose to add two new
allowed values for the basic keyword BITPIX. The set of values specified
in the original FITS Agreement was 8, 16 and 32; we propose to add -32 and
-64 to indicate IEEE single- and double-precision floating point data.
Thus, the number of bits per pixel would be the absolute value of BITPIX.
This implies that the number of 2880-byte logical records used by the main
data matrix must be computed using the absolute value of BITPIX (this rule
was specified in the Generalized Extensions Agreement (Grosb{\o}l etal.
1988)).
The IEEE standard specifies a variety of special exponent and mantissa
values in order to support the concepts of plus and minus infinity, plus
and minus zero, 'denormalized' numbers and 'not-a-number' (NaN). We propose
that all of these special cases should be fully accepted for FITS
interchange.
We propose that the order of the bytes should be sign and exponent first,
followed by the mantissa bytes in order of decreasing significance (i.e.,
the standard non-byte-swapped order).
The BLANK keyword of the original FITS Agreement should be ignored by FITS
readers when BITPIX=-32 or -64 (the NaNs of the IEEE format will act as the
blank). FITS writers should not write the BLANK keyword if BITPIX=-32 or
-64.
The BSCALE and BZERO values should be applied by FITS readers if they
differ from 1.0 and 0.0. However, such usage be regarded as bad practice,
and strongly discouraged, because of the risk of generating overflows and
underflows in FITS readers.
\subsection{Bibliography}
"An Implementation Guide to a Proposed Standard for Floating-Point
Arithmetic", COMPUTER magazine, Vol. 13, No.  1, pp.68-79, January 1980.
COMPUTER, Vol. 14, No.  3, March 1981.
ANSI/IEEE  754-1985, "IEEE Standard for Binary Floating Point Arithmetic".
ISDN-xxxxx?
Wells, D.C., Greisen, E.W., Harten, R.H.: 1981, Astron. Astrophys. Suppl.
Ser. 44, 363, "FITS: A Flexible Image Transport System".
Greisen, E.W., Harten, R.H.: 1981, Astron. Astrophys. Suppl. Ser. 44, 371,
"An Extension of FITS for Groups of Small Arrays of Data".
Grosb{\o}l, P., Harten, R.H., Greisen, E.W., Wells, D.C.: 1988, Astron.
Astrophys. Suppl. Ser.  73, 359-364, "Generalized Extensions and Blocking
Factors for FITS".
Harten, R.H., Grosb{\o}l, P., Greisen, E.W., Wells, D.C.: 1988, Astron.
Astrophys. Suppl. Ser. 73, 365, "The FITS Tables Extension".
\subsection{Implementation Notes}
Note-1: Many data analysis systems today use floating point internally and
so conversion costs for FITS input and output will be reduced by this
proposal. For machines with IEEE hardware the conversion cost will be
nearly zero, especially when BZERO=0 and BSCALE=1 and the reader has two
DO-loops (one with and one without scaling).
Note-2: The interpretation of the IEEE sign, exponent and mantissa bit
fields is precisely specified by the five rules of Section 3.2.1 of
ANSI/IEEE Standard 754-1985. Algorithms for transformation between IEEE and
non-IEEE formats can be derived from these rules. The FITS Committees will
provide a central archive for conversion software for various
architectures.
FITS readers on non-IEEE systems should test the IEEE exponents for
the special values 0 and 255/2047 (for 32/64-bit numbers). Exponent
zero should be mapped to the local value for floating zero (exponent
zero is used by IEEE for 'denormalized' numbers as well as for both
plus and minus zero); exponent 255/2047 should be mapped to the local
value used for blank pixels (all IEEE infinity and NaN cases have
exponent 255/2047). The IEEE exponent field can be selected from the
32/64-bit values by masking with hexadecimal byte string
7F,80,00,00/7F,F0,00,00,00,00,00,00 (i.e., the constant {\tt
7F800000}/{\tt 7FF0000000000000} hexadecimal on computers with the
normal word ordering; for VAXes use constant {\tt 00007F80}/{\tt
0000000000007FF0} instead). The two tests are made by comparing the
result against zero and against the mask.
FITS writers executing on non-IEEE systems should map their local
value for blank pixels to an IEEE NaN value; we suggest that all bits
set (minus one in 32- and 64-bit two-s complement integers) is a good
choice. The FITS writer should also map local floating point absolute
values which exceed the largest value of IEEE (approximately 10**38
for 32-bit, 10**304 for 64-bit) to IEEE-Infinity.  We recommend that
absolute values smaller than the smallest normalized IEEE value
(approximately 10**-38 or 10**-304) be mapped to zero. These two tests
are particularly important on machines which use 'hexadecimal'
normalization, with an exponent range of approximately 10**75 in
single-precision.
The DEC VAX architecture is an important special case. VAX systems use
the DEC-proprietary 'F' format for 32-bit floating data and their byte
order is 'swapped'. Once the swap has been corrected the F-format
differs from the IEEE 32-bit format only in that the exponent bias
differs by one and the (hidden) binary point is shifted by one bit.
The two differences add, and the total effect is that the two number
systems differ by a factor of 4.0 (VAXF = 4.0 * IEEE32, IEEE32 = 0.25
* VAXF). This floating point multiplication operation is little, if
any, more expensive than the conversion operation between integer and
floating point which is required by the original FITS Agreement. VAX
systems should test the exponent field on input to detect the infinity
and NaN cases and map them to the local blank code using the test
specified previously. The minus zero and denormalized cases may also
need to be trapped. Initial prototype implementations of this
Agreement have disclosed that some IEEE hardware produces -0.0 results
and that reserved operand errors occur in VAXes when these values are
used as operands. The 32-bit operand should be compared against 32768
to detect this case, and the result 0.0 be given rather than
multiplication by 4.0.
The DEC VAX architecture supports two different double precision
formats, 'D' and 'G'. The 'G' format has the same relationship to IEEE
64-bit format that the DEC 'F' format has to IEEE 32-bit (VAXG = 4.0 *
IEEE64). The 'D' format, however, requires a complicated
transformation to convert to/from IEEE 64-bit; programmers should
consult archives of FITS code to obtain suitable algorithms and
implementations for this case.
% ===================================================================
\clearpage
\section{Hierarchical Keywords in FITS Headers}\label{app:HKA}
\begin{center}P.~Grosb{\o}l and D.~Wells\end{center}
\begin{center}28 February 1989 (Draft-1)\end{center}
\subsection{The Proposal}
A new standard keyword HIERARCH\footnote{This name was derived by the
traditional FITS rule that names longer than 8 characters should be
truncated. It is, in fact, an English word in its own right: the {\bf
Oxford English Dictionary} says (definition B.2) that a {\em hierarch}
is the ``...commander of the celestial hierarchy''!} is introduced for
all FITS headers, both Basic FITS headers and Generalized Extension
headers. The remaining 72-characters of an 80-character HIERARCH line
will contain one or more keyword classification codes followed a
single keyword=value pair.  Code and keyword names will be 1-8
characters (uppercase alphanumeric plus the minus sign), just as for
standard keywords.  Column 9 will be blank; the first code will begin
in column 10 and additional codes and the keyword will be delimited by
one or more blanks.  Value fields will be decoded according to the
rules of Basic FITS.  The intent of this convention is to provide a
hierarchical classification of keywords in order to reduce the
possibility of conflicts of keyword names, especially 'proprietary'
keywords.  The set of approved top-level classification codes will be
controlled by the FITS Working Group of IAU Commission 5.
\subsection{Example}
\begin{verbatim}
         1         2         3         4         5         6
123456789012345678901234567890123456789012345678901234567890
HIERARCH topcode level2 code3 keyword = value
\end{verbatim}
\subsection{Discussion}
The IAU FITS Working Group will need to decide whether top-level
classification codes should be assigned for individual institutions
and projects, or for disciplines of astronomy.  Name-spaces assigned
to institutions can be managed privately by those institutions.
Name-spaces assigned to disciplines will require the creation of
international committees.  The discipline groups could assign
institutional codes under their top-level codes as well as agreeing on
discipline-specific generic codes and keywords.  The latter approach
can lead to a greater degree of standardization among special interest
groups in astronomy, without burdening the IAU Working Group.  This
is, in principal, very desirable.  However, it is unclear whether or
not such interest groups can be organized in practice.
\subsection{Notes}
\begin{enumerate}
\item The Basic-FITS rule that equal signs must be followed by one or
more blanks is relaxed for HIERARCH lines because the rules for
value-field syntax prevent ambiguities.
\item The maximum number of classification codes is naturally limited by
the finite length of a HIERARCH line; however, implementors should
probably limit HIERARCH codes to no more than five levels.
\item Note that Basic FITS does not specify whether keywords are limited
to the set of uppercase alphanumeric characters plus the minus sign,
but those are the only characters occurring in the examples given in
the 1981 paper.
\item Lines 2/4 and 2/5 of Figure 1 on page 370 of the Basic FITS paper
(Astron. Astrophys. Suppl. Ser. 44, 363-370, 1981) show an example of
a hierarchical keyword notation which is similar to the one proposed
here. It is regretable that this notation was not recommended more
clearly in the Basic FITS paper, as much confusion would have been
prevented over the years.
\end{enumerate}
% ===================================================================
\clearpage
\section{3-D Binary Table Extension Format}\label{app:C3DBTF}
% document translated from DEC RUNOFF to LaTeX format
% by program RNOTOTEX version CVF02B at 13-APR-1989 10:52:05.03
% Source file: CHAP14.RNO
%\chapter{FITS Tapes}
{\bf Note:} The text of this appendix is taken from Chapter~14 of {\bf
Going AIPS}, the AIPS programmer's reference manual. The chapter
describes how AIPS uses the FITS agreements. It gives an overview of
FITS from the AIPS perspective, and then proceeds to discuss each of
the FITS formats used by AIPS. Section~\ref{app:C3DBTF2} below (14.5.3
in {\bf Going AIPS}) describes the 3-D binary table format, which Bill
Cotton designed circa 1986 to meet the special requirements of the
calibration databases for the VLBA.
Subsections of the {\bf Going AIPS} chapter which are titled ``Image
Files'', ``Random Group (UV data) Files'', ``Older AIPS Tables'',
``AIPS FITS INCLUDEs'', and ``AIPS FITS Parsing Routines'' have been
deleted because they are not relevant to this paper.
The subsection titled ``Single Dish Data'' has been retained because
of its special interest for single-dish radio astronomers; it
describes the support provided by AIPS for forming N-dimensional
images out of signals (continuum or multi-spectral) sampled at {\em
random} locations by various scanning instruments.  This section shows
the random-groups approach to the spectroscopic format problem. It is
important to realize that any data structure which can be realized
within the random groups concept can also be realized within the 3-D
binary tables concept. Indeed, at the bit stream level the two formats
are essentially identical. The binary tables design does support
several data types which are not possible in random groups (the most
important case is character strings); this is a fundamental advantage
for the table approach.
\subsection{Overview }
The principal route for getting data and images into and out of AIPS
is by FITS (Flexible Image Transport System) format tape files. FITS
is an internationally adopted medium of exchange of astronomical data
and allows easy interchange of data between observatories and image
processing systems.  FITS also has the advantages that it is a
self-defining format and that the actual bit pattern on the tape is
independent of the machine on which the tape was written.  The purpose
of this chapter is to describe the general features of FITS and the
details of the AIPS implementation.  This chapter is not intended to
be a rigorous description of the FITS standards.  See the chapter on
devices for information on read and writing tapes in AIPS.
The fundamental definition of the FITS system\index{FITS format} is
given in Wells,
Greisen, and Harten (1981), with an extension described in Greisen and
Harten (1981).  A proposed further extension is given in Harten,
Grosb{\o}l, Tritton, Greisen and Wells, (1984).  FITS has been adopted as
the recommended medium of exchange of astronomical data by the IAU,
the Working Group on Astronomical Software (WGAS) of the AAS, and
comparable working groups in Europe.  AIPS now also supports the
proposal of these working groups for the writing of blocked FITS
tapes.\index{blocked tapes}
Because of the great flexibility of the FITS system, many of its
features have been adopted for the internal data storage format in
AIPS.  See the chapter on the catalog header for more details on the
AIPS internal storage format.
There are three main portions of a FITS file (1) the main header, (2)
the main data, and (3) any number of records containing auxiliary
information.  In addition, an extension of the original definition of
the FITS structure allows storage of ungridded visibility data.  Each
of these is discussed in detail in the following sections.
\subsection{Philosophy }
FITS is a philosophy as much as a data format.  The underlying
philosophy is to provide a standardized, simple, and flexible means to
transport data between computers or image processing systems. FITS is
standardized in the sense that any FITS reader should be able to read
any FITS image, at least to the degree that the array read is of the
correct dimensions and the pixel values have at least the correct
relative scaling.  In addition, any FITS reader should be able to cope
with any FITS format tape and, at least, skip over portions, or ignore
keywords, that it doesn't understand.
\index{FITS format}
The requirement of simplicity means that the implementation of FITS
reading and writing should be fairly straightforward on any computer
used for astronomical image processing.  Simple also implies that the
structure of the file should be self-defining and, to a large degree,
self-documenting.
The main advantage of FITS is its flexibility.  Due to the
self-defining nature of the files, a large range of data transport
needs are fulfilled. The introduction of new keywords gives the
ability to add new pieces of information as needed, and the use of
generalized extension files allows almost unlimited flexibility in the
type of information to be stored.  Thus, FITS can grow with the needs
of the Astronomical community.
The great flexibility of FITS is a potential weakness as well as a
strength.  There is a great temptation to proliferate keywords and new
extension file types. This should be done with great caution. Since
FITS is a worldwide medium of data exchange, there needs to be
coordination of keywords and extension files to prevent duplication
and inconsistencies in usage.
The most fundamental philosophical ideal of FITS is that no change in
the system should render old tapes illegal or unreadable. This
philosophy is reflected in the AIPS implementation of FITS in that all
obsolete implementations (e.g., old CLEAN component or antenna
extension files) are trapped and processed in the most accurate manner
possible.
\subsection{Single Dish Data }
     Observations made with filled aperature instruments are
frequently made at essentially random positions on the sky, possibly
using a number of offset feeds or detectors.  This type
of data may be convienently described using the random groups (UV
FITS) format.  The FITS form of this data is the same as visibility
data except that the number and meaning of the random parameters are
different.  The celestial coordinates may be either Right Ascension
and declination or projected coordinates about a specified tangent
point.
     A logical record consists of all data recorded from a given
beam on the sky at a given time.  A dummy AN table is optional.
\subsubsection{Single dish random parameters }
The single dish random parameter types (PTYPEn)
are described in the following:
\begin{enumerate} % list nest 1
\item 'RA' and 'DEC':  These random parameters are the Right
Ascension and Declination of the observation in degrees.  If the
coordinates have been projected onto the tangent plane then the RA and
Declination types become 'RA--xxxx' and 'DEC-xxxx' where xxxx is the
projection code.  See the chapter on AIPS catalogue headers and/or
AIPS memoes 27 and 46 for details of the projection codes.  These
random parameter these are required but the order is arbitrary.
\item 'DATE': The time tags for the data are kept in the form of Julian date
in days. This random parameter is required but the order is optional.
\item 'BEAM': This random parameter gives the beam number + 256. This random
parameter is optional.  The beam offset makes the data look more like
uv data and more of the the AIPS uv data tasks will work for this data.
\item 'SCAN': This random parameter gives the scan number. This random
parameter is optional.
\item 'SAMPLE': This random parameter gives the sample number in the scan.
This random parameter is optional.
\end{enumerate} % - list nest 1
\subsubsection{Single dish regular axis coordinates }
The units of the regular axis coordinates are defined by convention;
the conventions used by AIPS for the regular axis types (CTYPEn)
are the following:
\index{uv FITS format}
\begin{enumerate} % list nest 1
\item 'COMPLEX': the complex axis consists of the real, imaginary and
(optional) weight.  Magic value blanking is supported.  The imaginary
part may be used to carry any baseline values which have been
subtracted.  This axis is required.
\item 'STOKES': this axis is used to describe which Stokes' parameters are
given; the conventions are the same as used internally in AIPS.  These
conventions are discussed in the chapter on disk I/O.  This axis is
required.
\item 'FREQ': the frequency axis coordinates are in Hz.  This axis is required.
\item 'IF': The IF axis is a construct which allows irregularly spaced groups
of frequency channels.  The IF number specifies an entry in the ('AIPS
CH') table which must follow the data if this axis is present. This
table gives the offsets from the reference frequency specified by the
FREQ axis.  This axis is optional but if it is present then a CH table
must be present.
\item 'RA' and 'DEC': the celestial coordinates are given in degrees.  The
values associated with these axes are irrelevant (although they should
be present) for unprojected data.  For data with projected coordinates
the coordinate values of these axes should be the tangent point, i.e.
the position on the sky at which the plane onto which the coordinates
are projected is tangent to the celestial sphere and these axes should
become 'RA---ccc' and 'DEC--ccc' where ccc is the projection code.
These axes are required.
\end{enumerate} % - list nest 1
     Weights and flagging are handled the same as for visibility data.
Sort order is the same as for visibility data except that the sort
codes for sorting by u and v become:
\begin{verbatim}
          U => ordered by RA
          V => ordered by Declination
          X => descending ABS (RA)
          Y => descending ABS (Declination)
          Z => ascending ABS (RA)
          M => ascending ABS (Declination)
\end{verbatim}
\subsubsection{Example Single Dish Data File Header }
The following is a FITS header for a single dish data file
containing 16 frequency channels and a single IF and using unprojected
coordinated.  The first two lines are not part of the FITS file.
\index{single dish FITS format}
\begin{verbatim}
000000000111111111122222222223333333333444444444455555555556666666666
123456789012345678901234567890123456789012345678901234567890123456789
SIMPLE  =                    T /
BITPIX  =                   16 /
NAXIS   =                    6 /
NAXIS1  =                    0 /NO STANDARD IMAGE JUST GROUP
NAXIS2  =                    3 /
NAXIS3  =                    1 /
NAXIS4  =                   16 /
NAXIS5  =                    1 /
NAXIS6  =                    1 /
EXTEND  =                    T /TABLES FOLLOWING MAIN IMAGE
BLOCKED =                    T /TAPE MAY BE BLOCKED
OBJECT  =  'ALL SKY '          /SOURCE  NAME
TELESCOP=  '300 FT  '          /
INSTRUME=  '6 CM    '          /
OBSERVER=  'PHANTOM '          /
DATE-OBS=  '02/12/86'          /OBSERVATION START DATE DD/MM/YY
DATE-MAP=  '09/11/87'          /DATE OF LAST PROCESSING DD/MM/YY
BSCALE  =    4.98494155534E-05 /REAL = TAPE * BSCALE + BZERO
BZERO   =    0.00000000000E+00 /
BUNIT   =  'JY      '          /UNITS OF FLUX
EPOCH   =      1.950000000E+03 /EPOCH OF RA DEC
BLANK   =               -32768 /TAPE VALUE OF BLANK PIXEL
CTYPE2  =  'COMPLEX '          /
CRVAL2  =    1.00000000000E+00 /
CDELT2  =      1.000000000E+00 /
CRPIX2  =      1.000000000E+00 /
CROTA2  =      0.000000000E+00 /
CTYPE3  =  'STOKES  '          /
CRVAL3  =   -1.00000000000E+00 /
CDELT3  =     -1.000000000E+00 /
CRPIX3  =      1.000000000E+00 /
CROTA3  =      0.000000000E+00 /
CTYPE4  =  'FREQ    '          /
CRVAL4  =    1.66499989984E+09 /
CDELT4  =      5.000000000E+07 /
CRPIX4  =      1.000000000E+00 /
CROTA4  =      0.000000000E+00 /
CTYPE5  =  'RA      '          /
CRVAL5  =    0.00000000000E+00 /
CDELT5  =      1.000000000E+00 /
CRPIX5  =      1.000000000E+00 /
CROTA5  =      0.000000000E+00 /
CTYPE6  =  'DEC     '          /
CRVAL6  =    0.00000000000E+00 /
CDELT6  =      1.000000000E+00 /
CRPIX6  =      1.000000000E+00 /
CROTA6  =      0.000000000E+00 /
GROUPS  =                    T /
GCOUNT  =               14655. /
PCOUNT  =                    5 /
PTYPE1  =  'RA      '          /  RA in degrees
PSCAL1  =    1.09890000000E-02 /
PZERO1  =    0.00000000000E+00 /
PTYPE2  =  'DEC     '          /  Declination in degrees
PSCAL1  =    1.09890000000E-02 /
PZERO2  =    0.00000000000E+00 /
PTYPE3  =  'BEAM    '          /  Beam number
PSCAL3  =    3.66412235782E-10 /
PZERO3  =    0.00000000000E+00 /
PTYPE4  =  'DATE    '          /  Date + time as Julian date (days)
PSCAL4  =    2.50000000000E-01 /
PZERO4  =    2.44603650000E+06 /
PTYPE5  =  'DATE    '          /
PSCAL5  =    1.52587890600E-05 /
PZERO5  =    0.00000000000E+00 /
ORIGIN  = 'AIPSNRAO CVAX            15APR87'    /
DATE    = '13/11/87 '                    / FILE WRITTEN ON DD/MM/YY
HISTORY AIPS WTSCAL =  1.83759533423E+00 / CMPLX WTS=WTSCAL*
                                           (TAPE*BSCALE+BZERO)
END
\end{verbatim}
\subsection{Extension Files }
There is frequently auxiliary information associated with an image or
data set which needs to be saved in the same tape file. Examples of
this in AIPS are the Antenna files and CLEAN component files.  There
is currently a draft proposal to the IAU (Harten {\it et. al.} 1984)
defining a standard format for the invention of extension files to be
written after the main data records (if any) and defining a ``Tables''
type extension file.  The Tables extension files will be able to carry
information which can be expressed in the form of a table. AIPS also
makes use of a 3-D Table extension which is similar to tables but
allows arrays as table entries.  The following section will describe
the proposed standards which are being incorporated into AIPS.
\index{FITS format}
\index{FITS tables}
\subsection{Standard Extension }
The standard, generalized extension file is not a true tape file in
the sense that it is separated by tape EOF marks, but is a number of
records inside a FITS tape file which contains information of
relevance to the file.  Each standard extension ``file'' will have a
header which is very similar to the main FITS header.  This header
consists of one or more 2880 8-bit byte ``logical'' records each
containing 36 80-byte ``card images'' in the form:
\begin{verbatim}
     keyword = value    / comment
\end{verbatim}
The extension file header begins in the first record following the
last record of main data (if any) or following the last record of the
previous extension file.  The format of the generalized extension
``file'' header is such that a given FITS reader can decide if it wants
(or understands) a given extension file type and can skip over the
extension file if the reader decides it doesn't.
Most of the standards concerning data types and bit orders for the
main FITS data records also apply to extension files.  One difference
is that 8-bit pixel values can be used to indicate ASCII code.
The use of the generalized extension ``files'' requires the use of a
single additional keyword in the main header:
\begin{enumerate} % list nest 1
\item EXTEND (logical) if true (T) indicates that there {\it may} be extension
files following the data records and, if there are, that they conform
to the generalized extension file header standards.
\end{enumerate} % - list nest 1
\index{FITS format}
\index{FITS tables}
The required keywords in an extension file header record are, in
order:
\begin{enumerate} % list nest 1
\item XTENSION (character) indicates the type of extension file, this must
be the first keyword in the header.
\item BITPIX (integer) gives the number of bits per ``pixel'' value.  The
types defined for the main data records plus 8-bit ASCII are allowed.
\item NAXIS (integer) gives the number of ``axes''; a value of zero is allowed
which indicates that no data records follow the header.
\item NAXIS1 (integer) is the number of ``pixels'' along the first axis (if
any).
\item NAXISn (integer) is the number of ``pixels'' along the last axis.
\item PCOUNT (integer) is the number of ``random'' parameters before each
group.  This is similar to the definition of random group main data
records.  The value may be zero.
\item GCOUNT (integer) is the number of groups of data defined as for the
random group main data records. If an image-like file (e.g., a table
file) is being written this will be 1.
\item END is always the last keyword in a header.  The remainder of the
record following the END keyword is blank filled.
\end{enumerate} % - list nest 1
\index{FITS format}
\index{FITS tables}
There are three optional standard keywords for extension file header
records.  The order, between the required keywords and the END
keyword, is arbitrary.
\begin{enumerate} % list nest 1
\item EXTNAME (character) can be used to give a name to the extension file
to distinguish it from other similar files.  The name may have a
hierarchal structure giving its relation to other files (e.g.,
``map1.cleancomp'')
\item EXTVER (integer) is a version number which can be used with EXTNAME to
identify a file.
\item EXTLEVEL (integer) specifies the level of the extension file in a
hierarchal structure.  The default value for EXTLEVEL should be 1.
\end{enumerate} % - list nest 1
\index{FITS format}
\index{FITS tables}
The number of bits in an extension file (excluding the header) should
be given by the formula:
\begin{verbatim}
NBITS = BITPIX * GCOUNT * (PCOUNT + NAXIS1 * NAXIS2 * ... * NAXISn)
\end{verbatim}
The number of data records following the header record are then given
by:
\begin{verbatim}
     NRECORDS = INT ((NBITS + 23039) / 23040)
\end{verbatim}
It is important that the above formulas accurately predict the number
of data records in an extension ``file'' so that readers can skip over
these ``files''.  The data begins in the first record following the last
record of the header.
Extreme caution must be exercised when inventing new types of
extension files.  In particular, duplication of types, or several
types with the same function, must be avoided.  This means that when a
new extension file type is invented, it should be as general as
possible so that it may be used for other similar problems.
\subsection{Tables Extension }
A very common type of extension file is one containing data that can
be expressed in the form of a table.  That is, a number of entries
which are all identical in form.  A general, self-defining table
extension file type is proposed by Harten {\it et. al.} (1984). The
following sections describe the proposed format.
\index{FITS format}
\index{FITS tables}
The table extension file uses ASCII records to carry the tabular
information.  Each table entry will contain a fixed number of entries
(although the number can vary between different extension files). For
each entry is given (1) a label (optional), (2) the beginning column,
(3) an undefined value (optional), (4) a Fortran format to decode the
entry, (5) scaling and offset information (optional), and (6) the
units (optional).
\subsubsection{Tables Header Record }
The keywords for tables extension file headers are given in the
following:
\begin{enumerate} % list nest 1
\item XTENSION (character) is required to be the first keyword and has a
value 'TABLE   ' for table extension files.
\item BITPIX (integer) is a required keyword which must have a value of 8
indicating printable ASCII characters.
\item NAXIS (integer) is a required keyword which must have a value of 2 for
tables extension files.
\item NAXIS1 (integer) is a required keyword which gives the number of
characters in a table entry (i.e., ``row'').
\item NAXIS2 (integer) is a required keyword which gives the number of
entries in the table (i.e., number of rows). A value of 1 is allowed.
\item PCOUNT (integer) is a required keyword which must have the value of 0
for tables extension files.
\item GCOUNT (integer) is a required keyword which must have the value of 1
for tables extension files.
\item TFIELDS (integer) is a required keyword which must follow the GCOUNT
keyword.  TFIELDS gives the number of fields in each table entry.
\item EXTNAME (character) is the name of the table.
\item EXTVER (integer) is the version number of the table.
\item EXTLEVEL (integer) is the hierarchal level number of the table, 1 is
recommended. (optional)
\item TBCOLnnn (integer) the pixel number of the first character in the
nnn'th field.
\item TFORMnnn (character) the Fortran format of field nnn (I,A,E,D)
\item TTYPEnnnn (character) the label for field nnn. (optional, order
arbitrary)
\item TUNITnnn (character) the physical units of field nnn. (optional, order
arbitrary)
\item TSCALnnn (floating) the scale factor for field nnn. True\_value =
tape\_value $\ast$ TSCAL + TZERO.  Note: TSCALnnn and TZEROnnn are not
relevant to A-format fields. Default value is 1.0 (optional, order
arbitrary)
\item TZEROnnn (floating) the offset for field nnn. (See TSCALnnn.) Default
value is 0.0 (optional, order arbitrary)
\item TNULLnnn (character) the (tape) value of an undefined value. Note: an
exact left-justified match to the field width as specified by TFORMnnn
is required. (optional, order arbitrary)
\item AUTHOR (character) the name of the author or creator of the table.
(optional, order arbitrary)
\item REFERENC (character) the reference for the table. (optional, order
arbitrary)
\item END must always be the last keyword and the remainder of the record
must be blank filled.
\end{enumerate} % - list nest 1
\index{FITS format}
\index{FITS tables}
The TFORMnnn keywords should specify the width of the field and are of
the form Iww, Aww, Eww.dd, or Dww.dd (integers, characters, single
precision and double precision).  If -0 is ever to be distinguished
from +0 (e.g., degrees of declination), the sign field should be
declared to be a separate character field.
\subsubsection{Table Data Records }
The table file data records begin with the next record following the
last header record and each contains 2880 ASCII characters in the
order defined by the header.  Table entries do not necessarily begin
at the beginning of a new record.  The last record should be blank
filled past the end of the valid data.
\subsubsection{Example Table Header and Data }
The first two lines of numbers are only present to show card columns
and are not part of the extension file.
\index{FITS format}
\index{FITS tables}
\begin{verbatim}
         1         2         3         4         5         6         7
12345678901234567890123456789012345678901234567890123456789012345678901
XTENSION= 'TABLE   '           / EXTENSION TYPE
BITPIX  =                    8 / PRINTABLE ASCII CODES
NAXIS   =                    2 / TABLE IS A MATRIX
NAXIS1  =                   60 / WIDTH OF TABLE IN CHARACTERS
NAXIS2  =                  449 / NUMBER OF ENTRIES IN TABLE
PCOUNT  =                    0 / RANDOM PARAMETER COUNT
GCOUNT  =                    1 / GROUP COUNT
TFIELDS =                    3 / NUMBER OF FIELDS IN EACH ROW
EXTNAME = 'AIPS CC '           / AIPS CLEAN COMPONENTS
EXTVER  =                    1 / VERSION NUMBER OF TABLE
TBCOL1  =                    1 / STARTING CHAR. POS. OF FIELD N
TFORM1  = 'E15.6   '           / FORTRAN FORMAT OF FIELD N
TTYPE1  = 'FLUX    '           / TYPE (HEADING) OF FIELD N
TUNIT1  = 'JY      '           / PHYSICAL UNITS OF FIELD N
TSCAL1  =                  1.0 / SCALE FACTOR FOR FIELD N
TZERO1  =                  0.0 / ZERO POINT FOR FIELD N
TBCOL2  =                   17 / STARTING CHAR. POS. OF FIELD N
TFORM2  = 'E15.6   '           / FORTRAN FORMAT OF FIELD N
TTYPE2  = 'DELTAX  '           / TYPE (HEADING) OF FIELD N
TUNIT2  = 'DEGREES '           / PHYSICAL UNITS OF FIELD N
TSCAL2  =                  1.0 / SCALE FACTOR FOR FIELD N
TZERO2  =                  0.0 / ZERO POINT FOR FIELD N
TBCOL3  =                   33 / STARTING CHAR. POS. OF FIELD N
TFORM3  = 'E15.6   '           / FORTRAN FORMAT OF FIELD N
TTYPE3  = 'DELTAY  '           / TYPE (HEADING) OF FIELD N
TUNIT3  = 'DEGREES '           / PHYSICAL UNITS OF FIELD N
TSCAL3  =                  1.0 / SCALE FACTOR FOR FIELD N
TZERO3  =                  0.0 / ZERO POINT FOR FIELD N
END
\end{verbatim}
The rest of the header block is blank filled.  The data cards start on
the next block boundary.
\begin{verbatim}
   0.183387E+00   -0.138889E-03    0.694444E-04
   0.146710E+00   -0.138889E-03    0.694444E-04
   0.117368E+00   -0.138889E-03    0.694444E-04
   0.938941E-01   -0.138889E-03    0.694444E-04
   0.183387E+00   -0.138889E-03    0.694444E-04
                      .
                      .
                      .
\end{verbatim}
\subsection{3-D Tables Extension }\label{app:C3DBTF2}
There are several types of extension files in AIPS which do not fit in
a natural way into the FITS Tables format (e.g., gain and calibration
tables).  These files tend to be large, and conversion to and from
ASCII can be expensive.  A ``3-D'' extension to the FITS tables format,
in which an entry can be a 1-dimensional array, gives the needed
flexibility.  Use of binary data, including IEEE floating point
formats, allows efficient implementation of readers and writers.
\index{FITS format}
\index{FITS tables}
Each row in a 3-D table contains a fixed number of entries
(although the
number can vary between different extension files).  The header is a
standard FITS extension header; for each table entry is given (1) the
size and data type of the entry, (2) a label (optional), (3) the units
(optional), and (4) an undefined value (optional).  An entry may be
omitted from the table, but still defined in the header, by using a
zero repeat count in the TFORMn entry.
\subsubsection{3-D Tables Header Record }
The keywords for 3-D tables extension file headers are given in the
following:
\begin{enumerate} % list nest 1
\item XTENSION (character) is required to be the first keyword and has a
value 'A3DTABLE' for table extension files.
\item BITPIX (integer) is a required keyword which must have a value of 8.
\item NAXIS (integer) is a required keyword which must have a value of 2 for
3-D Tables extension files.
\item NAXIS1 (integer) is a required keyword which gives the number of 8-bit
bytes in a table row.
\item NAXIS2 (integer) is a required keyword which gives the number of rows
in the table.  A value of 1 is allowed.
\item PCOUNT (integer) is a required keyword which must have the value of 0
for tables extension files.
\item GCOUNT (integer) is a required keyword which must have the value of 1
for tables extension files.
\item TFIELDS (integer) is a required keyword which must follow the GCOUNT
keyword.  TFIELDS gives the number of fields in each table entry.
\item EXTNAME (character) is the name of the table.
\item EXTVER (integer) is the version number of the table.
\item EXTLEVEL (integer) is the hierarchal level number of the table, 1 is
recommended. (optional)
\item TFORMnnn (character) is the size and data type of field nnn. If repeat
count is absent, it is assumed to be 1.  A value of zero is allowed.
(required, order arbitrary)
\item TTYPEnnnn (character) the label for field nnn.  (optional, order
arbitrary)
\item TUNITnnn (character) the physical units of field nnn.  (optional,
order arbitrary)
\item TNULLnnn (character) the value of an undefined value.  These are only
for integer data types; for floating data types, the IEEE nan (not a
number) values are used.
\item AUTHOR (character) the name of the author or creator of the table.
(optional, order arbitrary)
\item REFERENC (character) the reference for the table.  (optional, order
arbitrary)
\item END must always be the last keyword and the remainder of the record
must be blank filled.
\end{enumerate} % - list nest 1
\index{FITS format}
\index{FITS tables }
The TFORMnnn keywords specify the width of the field and are of the
form rL, rX, rI, rJ, rA, rE, or rD (logical, bit, 16-bit integers,
32-bit integers, characters, single precision and double precision,
r=repeat count).  If -0 is ever to be distinguished from +0 (e.g.,
degrees of declination), the sign field should be declared to be a
separate character field.
\subsubsection{Table Data Records }
The 3-D table file binary data logical records begin with the next
record following the last header record and each contains 2880 8-bit
bytes in the order defined by the header.  Table entries do not
necessarily begin at the beginning of a new record.  The last record
should be zero filled past the end of the valid data.
\index{FITS format}
\index{FITS tables}
The data types are defined in the following list (r is the repeat
count):
\begin{enumerate} % list nest 1
\item {\it rL.} A logical value consists of an 8-bit ASCII ``T'' indicating true
and ``F'' indicating false.
\item {\it rX.} A bit array will start in the most significant bit of the byte
and the following bits in the order of decreasing significance in the
byte.  A bit array entry consists of an integral number of bytes with
trailing bits zero.
\item {\it rI.} A 16-bit integer is a two's complement integer with the bits
in decreasing order of significance.
\item {\it rJ.} A 32-bit integer is a two's complement integer with the bits
in decreasing order of significance.
\item {\it rA.} Character strings are 8-bit ASCII characters in their natural
order.
\item {\it rE.} Single precision floating point values are in IEEE 32-bit
precision format with the bits in the following order:
\begin{verbatim}
                  1         2         3
        01234567890123456789012345678901
        seeeeeeeemmmmmmmmmmmmmmmmmmmmmmm
sign = -1 $\ast\ast$ s, exponent = eee..., mantissa = 1.mmmmm... .
The value is given by:
     value = sign * 2 **(exponent-127) * mantissa
\end{verbatim}
Note:  these values have a ``hidden'' bit and must always be normalized.
The IEEE nan (not a number) values are used to indicate an invalid
number; a value with all bits set is recognized as a nan.  The IEEE
special values (-0., +/- Infinity) are not recognized.  A
multiplication by a factor of 4.0 converts between VAX F and IEEE
32-bit formats.
\item {\it rD.}  Double precision floating point values are in IEEE 64-bit
precision format with the bits in the following order:
\begin{verbatim}
               1         2         3
     01234567890123456789012345678901
     seeeeeeeeeeemmmmmmmmmmmmmmmmmmmm
     3       4         5         6
     23456789012345678901234567890123
     mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm
sign = -1 ** s, exponent = eee..., mantissa = 1.mmmmm... . The value
is given by:
     value = sign * 2 **(exponent-1023) * mantissa
\end{verbatim}
Note: these values have a ``hidden'' bit and must always be normalized.
The IEEE nan (not a number) values are used to indicate an invalid
number; a value with all bits set is recognized as a nan. The IEEE
special values (-0., +/- Infinity) are not recognized. A
multiplication by a factor of 4.0 converts between VAX G and IEEE
64-bit formats.
\end{enumerate} % - list nest 1
\index{FITS format}
\index{FITS tables}
\subsubsection{Example 3- D Table Header }
The first two lines of numbers are only present to show card columns
and are not part of the extension file.
\index{FITS format}
\index{FITS tables}
\begin{verbatim}
         1         2         3         4         5         6
1234567890123456789012345678901234567890123456789012345678901234
XTENSION= 'A3DTABLE'           / EXTENSION TYPE
BITPIX  =                    8 / BINARY DATA
NAXIS   =                    2 / TABLE IS A MATRIX
NAXIS1  =                   12 / WIDTH OF TABLE IN BYTES
NAXIS2  =                  449 / NUMBER OF ENTRIES IN TABLE
PCOUNT  =                    0 / RANDOM PARAMETER COUNT
GCOUNT  =                    1 / GROUP COUNT
TFIELDS =                    3 / NUMBER OF FIELDS IN EACH ROW
EXTNAME = 'AIPS CC '           / AIPS CLEAN COMPONENTS
EXTVER  =                    1 / VERSION NUMBER OF TABLE
TFORM1  = '1E      '           / COUNT AND DATA TYPE OF FIELD N
TTYPE1  = 'FLUX    '           / TYPE (HEADING) OF FIELD N
TUNIT1  = 'JY      '           / PHYSICAL UNITS OF FIELD N
TFORM2  = '1E      '           / COUNT AND DATA TYPE OF FIELD N
TTYPE2  = 'DELTAX  '           / TYPE (HEADING) OF FIELD N
TUNIT2  = 'DEGREES '           / PHYSICAL UNITS OF FIELD N
TFORM3  = '1E      '           / COUNT AND DATA TYPE OF FIELD N
TTYPE3  = 'DELTAY  '           / TYPE (HEADING) OF FIELD N
TUNIT3  = 'DEGREES '           / PHYSICAL UNITS OF FIELD N
END
\end{verbatim}
The rest of the header block is blank filled. The binary data starts
on the next logical block boundary. The last record of table data is
zero filled past the end of the valid data.
\subsection{References }
\hspace*{  0.00cm } Wells, Greisen, and Harten 1981, {\it Astronomy and Astrophysics}
Supplement series, vol. 44, pp 363 - 370.
\smallskip
\hspace*{  0.00cm } Greisen and Harten, 1981, {\it Astronomy and Astrophysics}  Supplement
Series, vol. 44, pp 371 - 374.
\smallskip
\hspace*{  0.00cm } Harten, Grosb{\o}l, Tritton, Greisen and Wells 1984, draft reproduced in
the IAU Commission 9 {\it Astronomical Image Processing Circular.}
Also appeared in {\it Mem. S. A. It.}, 56, 437 (1985).
\end{document}
From dwells@fits.CX.NRAO.EDU Tue Oct 17 10:41:30 1989
Received: from NRAO.EDU (cv3.cv.nrao.EDU) by fits.cx.nrao.edu (4.0/SMI-DDN)
	id AA07359; Tue, 17 Oct 89 10:41:27 EDT
Received: from fits.cx.nrao.edu by NRAO.EDU (3.2/DCW-2n    )
	id AA12333; Tue, 17 Oct 89 10:41:00 EDT
Received: by fits.cx.nrao.edu (4.0/SMI-DDN)
	id AA07347; Tue, 17 Oct 89 10:09:35 EDT
Return-Path: <dwells>
Message-Id: <8910171409.AA07341@fits.cx.nrao.edu>
Status: RO
From: dwells@fits.CX.NRAO.EDU (Don Wells)
Sender: dishfits-request@fits.CX.NRAO.EDU
To: dishfits@fits.CX.NRAO.EDU
Cc: dwells@fits.CX.NRAO.EDU, hanisch@stsci.edu
Subject: Re:  latest proposal -- some comments and questions
Date: Tue, 17 Oct 89 10:09:27 EDT
Friends, Bob Hanisch sent me this last night:
BH>Date: Mon, 16 Oct 89 17:51:42 EST
BH>From: (Bob Hanisch) <hanisch@stsci.edu>
BH>Subject: latest proposal -- some comments and questions
BH>
BH>Hi Don.  Have been reading through your proposal that came out on DISH-FITS
BH>today, and am puzzled by a few things.
BH>
BH>First, I guess this is not really a puzzle, but just a comment.  At various
BH>times you equivalence columns in a binary table to keywords, and vice versa.
BH>In Section 3.1, a keyword is taken as a virtual column.  And in Section 3.5
BH>columns in the table are consider as virtual keywords.  I don't really see
BH>any difficulty here, except perhaps that this equivalence, or maybe a better
BH>term would be `correspondence', be discussed more explicitly.
You are right. These two sections should be merged in the next revision of
the document (I did mark it as "{\em DRAFT}" and gave the date as "\today").
Yes, "correspondence" is the right word.
BH>There is a
BH>good analogy in SDAS format data, and for that matter, in FITS group format.
BH>There are _global_keywords_, which appear in the main header, and _subset_
BH>specific_ keywords, which appear encoded in the group parameter block.  The
BH>STSDAS/IRAF system regards both as keywords, period, and values are accessed
BH>via the name of the keyword.  It's just that the latter keywords are multi-
BH>valued within the same data file, whereas the former are fixed.
BH>
BH>I don't understand Section 3.3, specifically, "the order be reversed so that
BH>the last substring, the keyword, will be in the first 8 characters."  I could
BH>not find any examples of what you meant, but if I take you at your word then
BH>the standard header for a FITS image might have something like this
BH>
BH>WHATEVER= value NRAO-12M GENERAL SINGLDSH / here is some comment info
BH>
BH>while the same thing, written in an extension header, would be
BH>
BH>SINGLDSH GENERAL NRAO-12M WHATEVER= value / here is some comment info
BH>
BH>Is this what you mean?  I'll reserve comments until I understand your intent.
Sorry for the confusion -- I will have to find better words in the next
draft, and add examples. What I meant was that in any header, main or
extension, we would have (in the HIERARCH notation)
HIERARCH SINGLDSH GENERAL NRAO-12M WHATEVER= value / here is some comment info
but that if the same keyword is a *column heading* in a 3-D table, then it
should appear as
TTYPE13 = 'WHATEVER NRAO-12M GENERAL SINGLDSH HIERARCH' / some comment info
There are two reasons for this suggestion: (1) the classic rule that string
values be unique/useful in first 8 characters and (2) implementors of
routines to display tables will want to be able to truncate the label
strings.
BH>On the more general question of using binary tables for single-dish data,
BH>I don't really have any reservations that this is indeed the most efficient
BH>way to go.  It is logically straightforward, relatively simple to implement,
BH>and clearly the most efficient thing to do within the context of FITS.  As
BH>has come up in other discussions, however, most notably in Ann Arbor, I think
BH>we need to generalize binary tables to include multi-dimensional arrays
BH>within the structure.  This seems to me to be such an obvious extension,
BH>I'm a bit surprised that Bill did not encounter it in dealing with VLA
BH>and VLBA data.
OK, OK, because I wanted to sell this idea to spectroscopists (1-D) I didn't
tell them how I intended to argue in the task force session where this would
be settled. I really think that, consistent with 10 years of FITS
traditions, we should do an N-dimensional solution if we can. In my section
5 where I use TTYPEnn values of 'NCHAN' and 'SPECTRUM', please read 'NAXIS',
'NAXIS1' and 'MATRIX', except that they would really appear as:
HIERARCH SINGLDSH GENERAL NAXIS = 1		/ 1-D spectrum (virtual column)
TTYPE5  = 'NAXIS1  SINGLDSH GENERAL HIERARCH'	/ (NCHAN)
TTYPE6  = 'MATRIX  SINGLDSH GENERAL HIERARCH'	/ (SPECTRUM)
I trust that the generalizations are obvious. An interesting side-effect of
this concept is that because the maximum product of the dimensions is set by
TFORM6 = '1024F', if NAXIS is 2 or greater this design allows matricies of
variable aspect ratio in the table! Note that value strings should be padded
with blanks in the first eight columns so that the first classification code
substring appears in either column 9 (as shown above) or in column 10 (if
the keyword is 8 characters long).
BH>The thing I was confused about in your discussion of binary tables, however,
BH>was in Section 3.2 - Multiple Feeds.  Aren't you overlooking being able to
BH>use the prime advantage of binary tables, i.e., table columns that are arrays,
BH>by saying that multiple feed data or multi-band data should go into separate
BH>structures?  I mean, if I have some feed- or band- dependent keywords, which
BH>I wish to store in a binary table, can't I define keyword columns of depth
BH>equal to the number of feeds or bands?  In a simple example, suppose I have
BH>four feeds, and one feed-dependent keyword.  Then the binary table could look
BH>like
BH>
BH>keyword(4I)	feed1(100E)	feed2(100E)	feed3(100E)	feed4(100E)
BH>
BH>Element 1 of the keyword array corresponds to feed1, element 2 to feed 2, etc.
BH>I imagine this may violate some basic tenet of relational databases, right?
BH>This is not an area of my expertise.  Of course, another thing I'd like to do
BH>is write this as
BH>
BH>keyword(4I)	feed(4x100E)
BH>
BH>which seems a lot cleaner.  Actually, probably want (100Ex4).
Yes, what you say is the alternative concept. Because I am inexperienced in
relational database design I hesitate to use RDBMS arguments one way or the
other. I merely think that it will be easier to build the spectral analysis
programs and easier to negotiate in the task force if we instead put the
four feeds in four separate rows of the table. I.e., I think that both
design and negotiation will be easier if there is only one 'MATRIX' column
in a table.  Anyway, if you take the "correspondence" principle seriously,
you would be arguing for multi-valued keywords, which the reviewers forced
us to give up in May 1979.
BH>Anyway, I don't know if I'm the only one confused or not.  One thing we clearly
BH>agree about is that the hierarchical keyword situation needs to be resolved
BH>regardless of how the data is actually structured.  And by the way, don't
BH>misconstrue my comments in my memo about the stack approach to hierarchies,
BH>that is about whether we need hierarchies at all.  I think we do; all I wanted
BH>to do was get people to agree to this, and present the arguments I could think
BH>of against it.
I understood that and, as I said yesterday, I thought you did the community
a service by raising the issue. I too hope that they will agree with us that
we simply must control our namespace if we want to transmit meanings
reliably.
BH>Should be a fun meeting.
BH>
BH>Cheers, Bob
Regards, Don
From dishfits-request@fits.CX.NRAO.EDU Thu Oct 19 02:18:03 1989
Received: from NRAO.EDU (cv3.cv.nrao.EDU) by fits.cx.nrao.edu (4.0/SMI-DDN)
	id AA01025; Thu, 19 Oct 89 02:18:01 EDT
Received: from fits.cx.nrao.edu by NRAO.EDU (3.2/DCW-2n    )
	id AA16752; Thu, 19 Oct 89 02:17:59 EDT
Received: by fits.cx.nrao.edu (4.0/SMI-DDN)
	id AA01022; Thu, 19 Oct 89 02:12:25 EDT
Return-Path: <NSSDCA::PSI%RPEPPING::RNORRIS@cvax.CV.NRAO.EDU>
X-Vms-Mail-To: FITS
Message-Id: <891019013207.00002B27061@cvax.CV.NRAO.EDU>
Status: RO
From: "Ray Norris, Australia Telescope, CSIRO, Australia" <NSSDCA::PSI%RPEPPING::RNORRIS@NRAO.EDU>
Sender: dishfits-request@fits.CX.NRAO.EDU
To: dishfits@fits.CX.NRAO.EDU
Subject: Use of FITS for real-time data logging
Date: Thu, 19 Oct 89 01:32:07 EDT
Gday all
Since nobody from the Australia Telescope will be present at the
Greenbank meeting, I'll be sending a written presentation instead.
Before I do so however, I'd like to invite comments on an issue
which hasn't so far been raised explicitly in this particular forum.
The issue is this: we would like to use FITS for logging data in real time,
thus avoiding the overhead of transcribing, and minimising the
proliferation of formats. But FITS generally requires the number of data
to be entered in some form (e.g. GCOUNT, NAXISn) in a header prior to
writing the data. However, in real life when observing you don't know in
advance how much data you'll be writing. You may, for example, plan to
do 100 integrations on a source, but then have to abort the observation half
way through because of high wind, a hardware failure, or simply because
the on-line monitoring tells you it's not worth continuing. So, you want
to record the existing 50 integrations. If you are using FITS on-line,
the ways of dealing with this situation are:
i) You record the data on disk in an intermediate format, and only write
it to tape at the end of the observation. But this defeats the point
of using FITS as an on-line format. You're effectively using the
intermediate format as your on-line format, and then later using
valuable cpu cycles to transcribe the data into FITS.
ii) You record each integration as a seperate FITS file. This works, but
(a) you end up with a tape full of small files - messy!, and (b) a large
fraction of the tape will contain redundant header information.
iii) You argue that it's a rare event not to finish the scan, and in those
rare instances you simply pad out the FITS file with blanked data so that
the data match the header. But this is impractical if you've started a
6-hour scan and decide to abort after one hour. Do you throw away the
1 hour of data or do you watch your tape going round for the next few hours?
This problem has been the principal reason that the AT is now using
(yet another) adhoc format (RPFITS) rather than true FITS for data logging.
This is a shame, as the use of standard FITS would be a much more
elegant solution, and would remove the need for subsequent transcription.
I would like to invite comments on the proposal that this problem
could be circumvented by allowing GCOUNT etc to be set to -1, which
means that the length of data is unknown. In this case, any FITS-reading
program skipping through the data will simply
have to read laboriously through until it sees a block of data
starting with a recognisable keyword (this approach is used by RPFITS
and it works). Of course for normal data interchange between sites
a positve value of GCOUNT would be entered as normal. This extension
of FITS can therefore sit quite happily alongside the existing FITS files,
but opens up FITS to be useful for on-line logging as well as its
original function of data interchange.
Regards,
Ray Norris
From dishfits-request@fits.CX.NRAO.EDU Thu Oct 19 09:03:21 1989
Received: from NRAO.EDU (cv3.cv.nrao.EDU) by fits.cx.nrao.edu (4.0/SMI-DDN)
	id AA01411; Thu, 19 Oct 89 09:03:19 EDT
Received: from fits.cx.nrao.edu by NRAO.EDU (3.2/DCW-2n    )
	id AA17509; Thu, 19 Oct 89 09:03:09 EDT
Received: by fits.cx.nrao.edu (4.0/SMI-DDN)
	id AA01357; Thu, 19 Oct 89 08:57:21 EDT
Return-Path: <POSTMASTER@NRAO.EDU>
X-Vms-Mail-To: DISHFITS AT NRAO
Message-Id: <891019084303.000033B9031@cvax.CV.NRAO.EDU>
Status: RO
From: "GUILLOTE@FRGAG51" <POSTMASTER@NRAO.EDU>
Sender: dishfits-request@fits.CX.NRAO.EDU
To: dishfits@fits.CX.NRAO.EDU
Subject: Ray Norris issue
Date: Thu, 19 Oct 89 08:43:03 EDT
Date sent:  19-OCT-1989 11:10:03.18
To:   DISHFITS@NRAO
Here are my own comments on Ray Norris issue of using FITS as an on-line
data format.
First general remark :
FITS is a Flexible Image TRANSPORT System, NOT an all purpose data
format. Use it for what it is.
RN> If you are using FITS on-line, the ways of dealing with this situation are:
RN> i) You record the data on disk in an intermediate format, and only write
RN> it to tape at the end of the observation. But this defeats the point
RN> of using FITS as an on-line format. You're effectively using the
RN> intermediate format as your on-line format, and then later using
RN> valuable cpu cycles to transcribe the data into FITS.
Comment :
Cpu cycles concern only the Header, not the data because here a mere
copy is required. But you are losing many more cpu cycles in your
data reduction system formatting and unformatting the FITS header
later on. Are they less valuable ?
RN> ii) You record each integration as a seperate FITS file. This works, but
RN> (a) you end up with a tape full of small files - messy!, and (b) a large
RN> fraction of the tape will contain redundant header information.
Comment :
This is not efficient, but it works. You could add a packing program
later in your data processing stream.
RN> iii) You argue that it's a rare event not to finish the scan, and in those
RN> rare instances you simply pad out the FITS file with blanked data so that
RN> the data match the header. But this is impractical if you've started a
RN> 6-hour scan and decide to abort after one hour. Do you throw away the
RN> 1 hour of data or do you watch your tape going round for the next few hours?
No Comment
RN> This problem has been the principal reason that the AT is now using
RN> (yet another) adhoc format (RPFITS) rather than true FITS for data logging.
RN> This is a shame, as the use of standard FITS would be a much more
RN> elegant solution, and would remove the need for subsequent transcription.
RN> I would like to invite comments on the proposal that this problem
RN> could be circumvented by allowing GCOUNT etc to be set to -1, which
RN> means that the length of data is unknown. In this case, any FITS-reading
RN> program skipping through the data will simply
RN> have to read laboriously through until it sees a block of data
RN> starting with a recognisable keyword (this approach is used by RPFITS
RN> and it works). Of course for normal data interchange between sites
RN> a positive value of GCOUNT would be entered as normal. This extension
RN> of FITS can therefore sit quite happily alongside the existing FITS files,
RN> but opens up FITS to be useful for on-line logging as well as its
RN> original function of data interchange.
Comment :
Your problem is not specifically related to FITS. It happens in
any acquisition system in which the data size must be written
but is not a priori predictable. I believe you missed the "classical",
easy to implement, yet standard and general solution, because you
reason in terms of TAPE (which is a purely sequential device) instead of FILE.
If you write your data on DISK first instead of directly on tape,
then it is easy to update the first 2880 bytes FITS block putting
the right GCOUNT when the scan finishes. The tape copy would be
done only when the file is closed. If you want to be safe against
computer crashes, you could update this header each time a new
information is written. I assume that your disk drives and system
are reliable enough, so that real-time tape copy is not absolutely
necessary.
There is no need to change anything in the FITS standard to answer
your request.
----------------------------------------------------------------------
A general, and personal, comment on FITS :
The success of FITS is largely due to its virtue of STANDARD, and
its original simplicity. Some people would argue that it is not
Standard or simple enough, and propose RITS (RIGID Image Transport
System) instead. I do not follow this opinion: up to now,
no unnecessary feature has been added. Instead, I believe we
should be carefull not to let FITS become RITS (RANDOM Image
Transport System) !
Computing power is no longer a problem, but programming power is still
one !
        Stephane
From dwells@fits.CX.NRAO.EDU Thu Oct 19 10:30:45 1989
Received: from NRAO.EDU (cv3.cv.nrao.EDU) by fits.cx.nrao.edu (4.0/SMI-DDN)
	id AA01543; Thu, 19 Oct 89 10:30:43 EDT
Received: from fits.cx.nrao.edu by NRAO.EDU (3.2/DCW-2n    )
	id AA17732; Thu, 19 Oct 89 10:30:33 EDT
Received: by fits.cx.nrao.edu (4.0/SMI-DDN)
	id AA01518; Thu, 19 Oct 89 10:22:24 EDT
Return-Path: <dwells>
Message-Id: <8910191422.AA01512@fits.cx.nrao.edu>
Status: RO
From: dwells@fits.CX.NRAO.EDU (Don Wells)
Sender: dishfits-request@fits.CX.NRAO.EDU
To: "PSI%RPEPPING::RNORRIS"@nssdca.gsfc.nasa.gov
Cc: dishfits@fits.CX.NRAO.EDU
Subject: Re:  Use of FITS for real-time data logging
Date: Thu, 19 Oct 89 10:22:17 EDT
I would like to add one more comment to Stephane's response to Ray's
issue. That is that in addition to using disk as an intermediate
storage medium one can also use RAM. In the past this was not
practical because RAM was too expensive and data acquisition machines
never had enough RAM. This situation is now changed completely.
RAM is *very* cheap now, and current state-of-the-art real-time CPUs
can easily have large amounts of it. For example, the VLBA Correlator
has Motorola MVME147 CPUs. These are two VME slots wide (the latest
model of the 147 is only one slot wide) and have a 25 MHz 68030 (5 MIPS),
68882 FP coprocessor, Ether and SCSI controller, and 8 MB of RAM. NRAO
is able to buy these for US$4200. It would be easy to make a competitive
procurement of one or more VME cards of additional RAM if we needed it.
The point is that it is now easy to have I/O buffers that are quite large,
to organize data in memory and update the header just before writing to
the device. The data can be in files of a few megabytes, each with a
proper header. I am not necessarily saying that this approach is preferable
to constructing larger files on a scratch disk, as Stephane suggested,
I am merely pointing out that it is yet another option for designers.
A small technical point about using disk as a buffer is that it can
boost performance of a streaming tape device by avoiding stop-start
overhead at intermediate data rates. One must be careful that aggregate
bandwidth on the backplane can support concurrent I/O from device to RAM,
RAM to disk, disk to RAM, and RAM to tape. At sustained rates of 200 kB/s
this is probably not a problem (assuming independent DMA controllers),
somewhere above 500 kB/s it will start to be a problem.
-Don
From dishfits-request@fits.CX.NRAO.EDU Fri Oct 20 07:22:46 1989
Received: from NRAO.EDU (cv3.cv.nrao.EDU) by fits.cx.nrao.edu (4.0/SMI-DDN)
	id AA03649; Fri, 20 Oct 89 07:22:44 EDT
Received: from fits.cx.nrao.edu by NRAO.EDU (3.2/DCW-2n    )
	id AA20439; Fri, 20 Oct 89 07:22:38 EDT
Received: by fits.cx.nrao.edu (4.0/SMI-DDN)
	id AA03646; Fri, 20 Oct 89 07:12:39 EDT
Return-Path: <NSSDCA::PSI%RPEPPING::RNORRIS@cvax.CV.NRAO.EDU>
X-Vms-Mail-To: DISHFITS
Message-Id: <891020070256.000006B3031@cvax.CV.NRAO.EDU>
Status: RO
From: "Ray Norris, Australia Telescope, CSIRO, Australia" <NSSDCA::PSI%RPEPPING::RNORRIS@NRAO.EDU>
Sender: dishfits-request@fits.CX.NRAO.EDU
To: dishfits@fits.CX.NRAO.EDU
Subject: Use of FITS for real-time data logging
Date: Fri, 20 Oct 89 07:02:56 EDT
Many thanks for your comments on real-time use of FITS. Before I
reply, I'd like to make a general observation. FITS has been
tremendously successful in easing the problems of data interchange,
and for that we must be eternally grateful to its originators. My
comments are not intended to point out deficiencies in FITS (in the
context of its original function) but to suggest a simple extension
which enables it to be used efficiently for an additional function:
that of data logging.
Like most observatories, we have guest observers who bring their own
favourite data reduction packages, or want to take the data back with
them. At present it is necessary to transcribe from one of a number
of 'internal' formats to FITS or some other 'external' format. We are
proposing to rationalise our internal formats, and in fact FITS with
the extension I propose makes for a very convenient internal format.
I agree that there is in principle no reason why you shouldn't go on
generating multitudes of various different ad hoc internal formats at
various observatories, except that it's very wasteful of precious
manpower in inventing the formats, writing transcription programs,
and transcribing data.
Having said all that, my specific comments on your replies are as
follows:
stephane> FITS is a Flexible Image TRANSPORT System, NOT an all purpose data
stephane> format. Use it for what it is.
Sorry, I disagree. It could, with a small modification, be even more
useful.
RN> i) You record the data on disk in an intermediate format, and only write
RN> it to tape at the end of the observation. But this defeats the point
RN> of using FITS as an on-line format. You're effectively using the
RN> intermediate format as your on-line format, and then later using
RN> valuable cpu cycles to transcribe the data into FITS.
stephane> Comment :
stephane> Cpu cycles concern only the Header, not the data because here a mere
stephane> copy is required. But you are losing many more cpu cycles in your
stephane> data reduction system formatting and unformatting the FITS header
stephane> later on. Are they less valuable ?
 I think we have to distinguish between two cases: (i) recording the
data  in one formnat, and then using an offline program to
transcribe, in  which case your comments apply, but see my comments
at the start of this mail.  (ii) using the observing program to
transcribe, in which case - yes cpu  cycles here are more valuable
than offline cycles! In the real-world the  real-time logging
computers tend to be under-powered, and tend not  to have the
number-crunching capacity of off-line processors, which can  always
spend a little longer than real time if a particular data set is
particularly voluminous.
RN> ii) You record each integration as a seperate FITS file. This works, but
RN> (a) you end up with a tape full of small files - messy!, and (b) a large
RN> fraction of the tape will contain redundant header information.
stephane>This is not efficient, but it works. You could add a packing program
stephane>later in your data processing stream.
 I agree - but's it's still messy and inefficient!
stephane>Your problem is not specifically related to FITS. It happens in
stephane>any acquisition system in which the data size must be written
stephane>but is not a priori predictable. I believe you missed the "classical",
stephane>easy to implement, yet standard and general solution, because you
stephane>reason in terms of TAPE (which is a purely sequential device) instead of FILE.
stephane>If you write your data on DISK first instead of directly on tape,
stephane>then it is easy to update the first 2880 bytes FITS block putting
stephane>the right GCOUNT when the scan finishes. The tape copy would be
stephane>done only when the file is closed. If you want to be safe against
stephane>computer crashes, you could update this header each time a new
stephane>information is written. I assume that your disk drives and system
stephane>are reliable enough, so that real-time tape copy is not absolutely
stephane>necessary.
Writing to disk is fine for most present systems, but becomes
impractical as data volumes get really large ( to take an extreme,
non-single-dish application, the AT produces data at a max rate of
about 1 Gbyte/hr. Future high-resolution multi-beam single dish
systems with rapid scanning may also approach this data rate). Here
you need either tape or optical disks, which are generally WORM, so
you can't easily overwrite the header. And you don't can't get round it
for WORM drives by leaving the writing of the header till later,
because then the system becomes fragile to machine crashes.
stephane>There is no need to change anything in the FITS standard to answer
stephane>your request.
I agree that there is no NEED to have a universal format - we can
certainly continue with a proliferation of ad hoc internal formats if
thats what the community wants. But the same arguments that led to
the birth of FITS can be extended: the astronomical community is
generally under-manned - why make life difficult when with a minor
change it could be so easy!
stephane>The success of FITS is largely due to its virtue of STANDARD, and
stephane>its original simplicity. Some people would argue that it is not
stephane>Standard or simple enough, and propose RITS (RIGID Image Transport
stephane>System) instead. I do not follow this opinion: up to now,
stephane>no unnecessary feature has been added. Instead, I believe we
stephane>should be carefull not to let FITS become RITS (RANDOM Image
stephane>Transport System) !
Yes but let's not allow it to become OITS (Obsolete Image Transport
System)!
In response to Don's comments about RAM etc, I guess we should look
very carefully at data rates. The AT rate cited above is an extreme
case, and may appear not to be relevant to the single-dish problem.
However, I believe we should be looking ahead, rather than coping
with problems that confront us NOW. Multi-feed,
high-spectral-resolution surveys may bring the hardware to its knees
(as Greenbank has already demonstrated - oops sorry - shouldn't have
said that!) and any standard designed now should be designed to last
for a few years.
To summarise, my arguments are that (i) to use FITS for on-line
logging makes sense in terms of efficient use of manpower and other
resources, (ii) it can be achieved with present FITS by some form of
buffering, but that is somewhat inefficient and may get impractical
as data rates increase, and (iii) allowing the option of GCOUNT=-1
has no effect whatsoever on the speed, efficiency, and implementation
of present FITS. You guys who don't like it can ignore it, and write
out GCOUNT as at present, EXCEPT that the FITS reading programs must
be extended to accomodate the possibility that they may encounter
data with GCOUNT=-1.
Cheers,
Ray Norris
From dishfits-request@fits.CX.NRAO.EDU Fri Oct 20 13:38:12 1989
Received: from NRAO.EDU (cv3.cv.nrao.EDU) by fits.cx.nrao.edu (4.0/SMI-DDN)
	id AA04196; Fri, 20 Oct 89 13:38:10 EDT
Received: from fits.cx.nrao.edu by NRAO.EDU (3.2/DCW-2n    )
	id AA21167; Fri, 20 Oct 89 13:37:55 EDT
Received: by fits.cx.nrao.edu (4.0/SMI-DDN)
	id AA04193; Fri, 20 Oct 89 13:28:24 EDT
Return-Path: <POSTMASTER@NRAO.EDU>
X-Vms-Mail-To: DISHFITS AT NRAO
Message-Id: <891020131623.00000CAF031@cvax.CV.NRAO.EDU>
Status: RO
From: "GUILLOTE@FRGAG51" <POSTMASTER@NRAO.EDU>
Sender: dishfits-request@fits.CX.NRAO.EDU
To: dishfits@fits.CX.NRAO.EDU
Subject: A comment on Don Wells reply to T.Forveille and S.Guilloteau proposal
Date: Fri, 20 Oct 89 13:16:23 EDT
Date sent:  20-OCT-1989 09:30:31.22
To:   DISHFITS@NRAO
Dear friends,
I have a comment on Don Wells answer to the our (T.Forveille and myself)
document. It concerns point 3) of his answer, which presumably was
corresponding to item (*5) (XCOORD for not regularly sampled data).
There has been here some confusion between "irregularly sampled"
and "randomly sampled" data, which originate from an incomplete
description in our document. In our point (*5), we were implicitely
refering to FREQUENCY axis, not to mapping coordinates. The frequency
axis may be irregularly sampled, (e.g. Acousto Optical Spectrometers
in radio telescopes) but this is usually in a systematic (not random!)
way. If one envisage to store part of a 3-D spectral mapping with such
a frequency axis, the use of a Table extension to indicate Frequency
Value versus Frequency Pixel is probably unavoidable (**). Using random
parameter instead would imply more or less doubling the data size
(more precisely adding (np-1)*nf values, where np is the number
of map points, and nf the number of frequency channels).
This is specially important not only for single-dish data, but also
for many telescopes with spectroscopic capabilities (HST, ISO, and
most optical telescopes (***).
                Stephane
(**) In fact this is similar (NOT identical) to the extra 'IF' axis, connected
to the 'AIPS CH' table, mentionned in Section C.3.2 of Don Wells proposal.
(***) And here comes again the question of Frequency versus Wavelength.
A regularly sampled axis in one domain is no longer regularly sampled
in the other.
From dishfits-request@fits.CX.NRAO.EDU Wed Oct 25 09:37:12 1989
Received: from NRAO.EDU (cv3.cv.nrao.EDU) by fits.cx.nrao.edu (4.0/SMI-DDN)
	id AA08577; Wed, 25 Oct 89 09:37:09 EDT
Received: from fits.cx.nrao.edu by NRAO.EDU (3.2/DCW-2n    )
	id AA04167; Wed, 25 Oct 89 09:36:41 EDT
Received: by fits.cx.nrao.edu (4.0/SMI-DDN)
	id AA08565; Wed, 25 Oct 89 09:26:11 EDT
Return-Path: <POSTMASTER@NRAO.EDU>
X-Vms-Mail-To: DISHFITS AT NRAO
Message-Id: <891025091334.000038C8031@cvax.CV.NRAO.EDU>
Status: RO
From: "PTW@STARLINK.RUTHERFORD.AC.UK" <POSTMASTER@NRAO.EDU>
Sender: dishfits-request@fits.CX.NRAO.EDU
To: dishfits@fits.CX.NRAO.EDU
Subject: comments on single-dish FITS proposals
Date: Wed, 25 Oct 89 09:13:34 EDT
Date sent:      Wed, 25 Oct 89  14:12 GMT
Received: from RL.IB by UKACRL.BITNET (Mailer X1.25) with BSMTP id 4118; Wed,
 25 Oct 89 14:11:09 BST
Received: from RL.IB by UK.AC.RL.IB (Mailer X1.25) with BSMTP id 9030; Wed, 25
          Oct 89 14:11:08 BS
Via:      UK.AC.RL.STAR; 25 OCT 89 14:11:00 BST
To:       DISHFITS@NRAO
Comments on single-dish FITS proposals
--------------------------------------
From:  P.T.Wallace
       Starlink Project
       Rutherford Appleton Lab
       Chilton, Didcot, Oxon OX11 0QX
       PTW @ STARLINK.RUTHERFORD.AC.UK
       19457::PTW
These are hurried comments based on one pass of the document.  Apologies
if I've misunderstood anything.
PHILOSOPHY
In our recent work on Starlink data formats, we have distinguished between
two different design philosophies.  The one we have adopted ourselves
attempts to describe the relationships between the different data
components sufficiently well for programs to process data from diverse
sources correctly.  The more common philosophy - in fact the only one
which astronomers believe in and think they understand - is what we call
the "thought of everything", or TOE, philosophy.  The Single dish format
seems to be an example of the latter.  The TOE idea is that once you have
given implementors enough pigeonholes into which to stuff the data
generated by their telescope and instrument, you have thereby defined
the "meaning" of everything in data processing terms.  I don't believe
it's that easy in practice.
This is not to say that it isn't possible to use such designs, which
at least mean that FITS readers can accept the contents of a tape, and
that simple utilities can do things like plotting pixel value against
x-coordinate.  But I don't believe an astronomer can do astrophysics
from a FITS tape without an accompanying letter describing how the
various components have been used.  To proceed rapidly to the point,
I would aim for greater rigor in some parts of the single-dish proposal,
and to move some of the present items out of the main format into
unashamedly site-specific extensions.
POINTING CORRECTIONS
I think it is a mistake to try and define a standard set of telescope
pointing corrections.  While it is true that any two-axis mount
will have just six numbers describing the purely geometrical terms
involved, and many people (myself included) like to base a pointing
model solidly on these six terms, it is of course possible to define
a pointing model in a various completely different ways - lookup tables,
polynomials or harmonics, etc.  Moreover, all telescopes will have
extra terms, and some may have auxiliary readings (from sensors mounted
on the structure) or quantities involved in attempts to deal with
hysteresis.  Also, I have on some telescopes found that there is no easy
way to get at the basic pointing information.  On the JCMT, for example,
the current pointing coefficients appear to be relative to whatever
values are embedded in the system, and, when pointing tests are done, no
attempt is made to log the raw encoder readings to allow a model to
be fitted ab initio.  (Not how I would do it.)  There are also the
fearful complications of subreflectors which move to correct for
deforming dishes, chopping secondaries, etc.
So I propose that all telescope model information be entirely site-
specific.
Also, I distrust the idea of user-supplied corrections, especially when
it isn't made clear whether the one in azimuth (etc) is dA or dA*cos(E).
I would rather leave this information in the (site-specific) telescope
model.
SIDEREAL TIME
It doesn't say whether this is mean or apparent.  I propose the
latter.
OMISSIONS
If one of the objectives is to be able to reconstruct the pointing,
consider including (i) station coordinates and (ii) UT-UTC.
The station coordinates should be longitude (east of Greenwich),
latitude (geodetic), and height above the current IAU reference
spheroid.  The longitude and latitude should, formally, be as
affected by polar motion.
ENVIRONMENT PARAMETERS
Does "index of refraction" (REFRAC) mean the refractive index,
or the tan Z refraction constant?  I'm not keen on the former, and
would supplement the latter with one more term (in tan**3 Z).
Is it necessary to have HUMIDITY and DEWPT and MMH20?  I would
go for HUMIDITY alone.
To calculate the radio/mm refraction properly (also important
at large Z in the optical case) you need the tropospheric lapse
rate, in deg K per metre.  You also need the height, which I have
included in my "station coordinates" suggestion.
SOURCE COORDINATES
There is some evidence of a confusion between the term "epoch"
(meaning a moment in time) and "equinox" (shorthand for "the
mean equator and equinox at a given epoch").  I suspect the
keyword SINGLDSH GENERAL EPOCH means equinox for example.
I recommend that the following coordinate systems are recognised:
a)  Pre-IAU-1976 mean RA,Dec for any given equinox and epoch.  The
equinox should be Besselian (and is usually 1950).  The epoch is
important (because the system is rotating relative to an inertial
frame).  E-terms are included (i.e. the quoted position embodies
the E-terms just like catalogue positions).  Recommended name "FK4".
b)  FK4 but without E-terms.  I suggest the name "BN", standing for
Bessel-Newcomb.
c)  Post-IAU-1976 mean RA,Dec for any given equinox and epoch.  The
equinox should be Julian (and is usually 2000).  The epoch is
not required unless you are going to bother with proper motions.
E-terms are not involved, of course.  Recommended name "FK5".
d)  Galactic coordinates.  The document could mention that you
should really calculate these from the "BN" B1950 position rather
than from "FK4".  Names: "L2B2", or "GAL" would be OK.
e)  Mean ecliptic and equinox of date, IAU 1976.  Recommended
name "ECL", or "MEAN_ECL" or something.
f)  Geocentric apparent RA,Dec, IAU 1976 + 1980.  Recommended
name "RADEC".  Note that "FK5" to "RADEC" consists of precession,
nutation, annual aberration, and the gravitational lens effect
of the Sun.
g)  Geocentric apparent HA,Dec.  Recommended name "HADEC".
Computed from "RADEC" by subtracting RA from LAST.
I am doubtful whether it is worth providing for topocentric
HA,Dec, which differs from "HADEC" by the diurnal aberration.
h)  Topocentric Az,El, obtained from topocentric HA,Dec by
rotating into Az,El using the long,lat as affected by polar
motion.  Recommended name "TOPO_AZEL".
i)  Observed Az,El, obtained from "TOPO_AZEL" by adding the
effects of atmospheric refraction.  Recommended name "OBS_AZEL".
j)  Raw Az,El, obtained from the telescope encoder systems.  This
is related to "OBS_AZEL" by the telescope pointing model.  Recommended
name "ENC_AZEL".
The proposal needs to specify for all Az/El items whether they
are raw, observed, or topocentric (and whether Az means Az or
dAz*cos(El)!).
-------------------------------------------------------------------------
From dishfits-request@fits.CX.NRAO.EDU Wed Oct 25 13:46:53 1989
Received: from NRAO.EDU (cv3.cv.nrao.EDU) by fits.cx.nrao.edu (4.0/SMI-DDN)
	id AA08871; Wed, 25 Oct 89 13:46:51 EDT
Received: from fits.cx.nrao.edu by NRAO.EDU (3.2/DCW-2n    )
	id AA04640; Wed, 25 Oct 89 13:44:10 EDT
Received: by fits.cx.nrao.edu (4.0/SMI-DDN)
	id AA08863; Wed, 25 Oct 89 13:34:54 EDT
Return-Path: <rww@hoh-2.att.com>
>From: hoh-2!rww (R. W. Wilson)
Message-Id: <8910251443.AA06406@orion.att.com>
Status: RO
From: rww@hoh-2.att.com
Sender: dishfits-request@fits.CX.NRAO.EDU
To: arpa!NRAO.edu!DISHFITS@hoh-2.att.com
Date: Wed, 25 Oct 89 10:43:42 EDT
Subject: Comments on a FITS Standard for Spectral Line Data
>From Robert W. Wilson, AT&T Bell Laboratories
Dear Single Dish FITS Conference Participants:
The most important thing, of course, is to have A STANDARD.  I will
almost certainly write a reader and writer for what ever new standard
emerges.
The one thing that I think must be changed from present FITS practice
is that multiple spectra should be allowed in a single file.  This
could be done as easily as simply writing multiple spectra one after
the other in a file.  It would take a minimum of cpu time to read the
header of an unwanted spectrum and calculate how many (2880 byte)
blocks to skip.  There would be no reason to announce how many spectra
are comming; the EOF would tell you when you are done.  This would
make exabyte tapes and disk files usable and all other efficiency
considerations would be of little consequence for the amount of data
that I foresee transporting in and out of Bell Labs with FITS.  (Tape
records blocked to a multiple [10?] of 2880 bytes would then be
desirable for IBM tapes.)
The other thing which is badly needed is a set of standard keywords.  I
think that there should be a required core set which would be adequate
for reduced data.  An extension set should be strongly encouraged if
the information is available and applicable.  As an example of what I
mean by this is in the data header which we have been using at the Bell
Labs 7-meter antenna.  A number of the parameters correspond to the
lock system which was used at the 36-foot in the early 1970's even
though our lock loop is different.  The only loss in keeping the old
definitions is a few arithmetic operations per spectrum in the
observing program (and occasionally puzzling over the code).  There was
considerable saving during the transition period when we had both kinds
of data.
I hope that I will not have to make use of very many telescope
dependent parameters when reading data from other observatories.
Observatories with idiosyncratic instruments should reduce their export
data so that it can be described in a normal way rather than expecting
all users to  add special code to handle their special case.  I include
in this category AOS spectrometers with non-linear frequency sampling.
If the original sampling is adequate, resampling on a regular grid
should not change the data (although some of the original high
frequency noise may be removed).  Of course transfer using FITS will
need to handle the raw data in whatever from it is produced for
internal transfer and transfer to frequent observers.
I like Bob Hanish's proposal for hierarchical keywords better than Don
Wells' proposal of stacking them up on the line with the most
particular word first.  I worry that this extensibility will be
abused.
I like the idea of 3-D binary tables for efficient transfer of large
amounts of data, but don't completely understand them yet.  If I am
forced to implement them, I might keep our raw data in that format,
since, except for the initial header the data file is already a binary
table.
As a final comment, I think that observers should be able to refer to
all of the data taken at the same time by a common observation number.
I am also used to integer 'scan' numbers and don't have a good proposal
for the case of multiple feeds/receivers/backends.  For exchange with
foreign systems, each spectrum should be a separate FITS entity with
some form of observation number in it.  (Actually this would argue
against keeping multiple backend raw data in FITS format.)
Bob Wilson
rww@hoh-2.att.com
201-888-7120
From dwells@fits.CX.NRAO.EDU Thu Oct 26 15:01:52 1989
Received: from NRAO.EDU (cv3.cv.nrao.EDU) by fits.cx.nrao.edu (4.0/SMI-DDN)
	id AA11069; Thu, 26 Oct 89 15:01:50 EDT
Received: from fits.cx.nrao.edu by NRAO.EDU (3.2/DCW-2n    )
	id AA07656; Thu, 26 Oct 89 15:01:40 EDT
Received: by fits.cx.nrao.edu (4.0/SMI-DDN)
	id AA11020; Thu, 26 Oct 89 14:54:19 EDT
Return-Path: <dwells>
Message-Id: <8910261854.AA11014@fits.cx.nrao.edu>
Status: RO
From: dwells@fits.CX.NRAO.EDU (Don Wells)
Sender: dishfits-request@fits.CX.NRAO.EDU
To: dishfits@fits.CX.NRAO.EDU
Subject: Hierarchical keywords
Date: Thu, 26 Oct 89 14:54:15 EDT
----- Begin Included Message -----
Date: Thu, 26 Oct 89 13:50:06 EDT
From: "ESOMC0::PREBEN"@cvax.CV.NRAO.EDU
Subject: Hierarchical keywords
To: dwells@fits.CX.NRAO.EDU
X-Vms-Mail-To: NRAO::DWELLS
Message-Id: <891026135006.00003FA4031@cvax.CV.NRAO.EDU>
                 Comments on Hierarchical Keywords in FITS
                -------------------------------------------
                               Preben Grosbol
                                1989-Oct-26
There are several reasons to consider a hierachical concept for keywords
in FITS. The main arguments for a more flexible syntax than the one
offered for basic FITS keywords are:
 1) A total of 8 characters is too limited to give a meaningfull
    description of the content.
 2) The small size of keywords increrase the chance of different
    institutes using the same keyword with different meaning.
 3) There are not clear definition of the name space for keywords which
    can avoid multi-defined keywords.
 4) The general problem of encoding small arrays as keywords is not solved.
A solution to the three first items was proposed by AIPS (using HISTORY
cards). It was refined with the concept of subject/group reserved
keywords like SNGLDISH for the Single Dish community. A more general
definition for a HIERARCH keyword was then put foreward based on the
AIPS and Single Dish experience.
The main argument against the HIERARCH is number of redundent characters
defining the high levels. This limits the usefull part of the 80 character
card significantly. The suggestion of Bob Hanish to introduce a HKBEGIN
and HKEND keywords to define the hierarchical level and use the basic keyword
syntax for the lowest level would solve the first requirement and make more
efficient usage of the limited space on a FITS keyword card. However, it
makes the present situation even worse for 2) and 3). It would allow
new low-level hierarchical keywords to be placed in column 1-8 where
old FITS readers would mix them with other simple non-standard FITS
keywords. It is MUST be ensured that additions to the FITS standard does
not generate wrong results even for old readers. This was the case with
the General Extensions, Blocking and Floating Point agreements.
Thus, the proposal of Bob Hanish for HKBEGIN/HKEND is NOT acceptable
due to the wrong interpretation of old FITS readers. The new concept
of hierarchical keywords MUST look like comments to old readers.
A number of options will provide acceptable solutions. Two main groups
could be discussed:
 1) The proposal already presented by Grosbol and Wells of type:
        HIERARCH level1 level2 level3 .. .. keyword = value / comment
 or
 2) A mix of this and the proposal of Hanish:
        HIERARCH BEGIN level1 level2 level3 .. ..
        HIERARCH keyword = value           / comment
        HIERARCH END
where the HIERARCH keyword can be exchanged with any other reasonable
combination of letters. The 'level1' identifiers must be controller
and should either be allocated to subject or groups.
With the choice of a special keyword (e.g. HIERARCH) one can upgrade
the syntax for the remaining of the card. It would be possible to
include a concept of small arrays in one of the following styles:
 1)  HIERARCH ARRAY(4) = 1, 2, 3, 4 / comment
    indicating an array with 4 elements in F77 list format of the type
    of the first element.
 2)  HIERARCH ARRAY(2..4) = 223.2, 33.3, -0.4 / comment
    indicating array elements 2, 3 and 4 in F77 list format of the type
    of the first element.
 3)  HIERARCH ARRAY(4I) = 1, 2, 3, 4 / comment
    indicating an array with 4 elements in F77 list format of integer type.
To provide space for small arrays a continue card would be needed. It
could be done by repeating the first keyword part (AIPS style) or
as the last non-comment character use a special sign e.g. '-'.
----- End Included Message -----
From dishfits-request@fits.CX.NRAO.EDU Thu Oct 26 16:47:02 1989
Received: from NRAO.EDU (cv3.cv.nrao.EDU) by fits.cx.nrao.edu (4.0/SMI-DDN)
	id AA11211; Thu, 26 Oct 89 16:47:00 EDT
Received: from fits.cx.nrao.edu by NRAO.EDU (3.2/DCW-2n    )
	id AA07936; Thu, 26 Oct 89 16:45:01 EDT
Received: by fits.cx.nrao.edu (4.0/SMI-DDN)
	id AA11208; Thu, 26 Oct 89 16:37:24 EDT
Return-Path: <olberg@oso.chalmers.se>
Message-Id: <8910261304.AA13092@oso.chalmers.se>
Status: RO
From: Michael Olberg <olberg@oso.chalmers.se>
Sender: dishfits-request@fits.CX.NRAO.EDU
To: dishfits@nrao.edu
Subject: Single-dish FITS
Date: Thu, 26 Oct 89 14:04:35 -0100
I would like to make a last minute contribution on two issues before the
workshop next week.
a) Assuming that a new FITS format would impliment a hierarchy of keywords
covering GENERAL, SNGLDISH GENERAL and finally SNGLDISH WHATEVER
keywords or anything equivalent to that, could we produce FITS reading
programs which could learn about anything that had not been general enough
to be implemented right from the start? I am thinking of something
like this:
Allow definition of FITS keywords within FITS:
0........1.........2.........3.........4..................................8
123456789012345678901234567890123456789012345.........................67890
DEFINE  = 'SEST-15M'           / start definition of SEST specific keywords
KEYWORD1= 'F10.1   '           / define KEYWORD1 and format how to read it
KEYWORD2= 'L10     '           / example of a keyword holding a logical ...
KEYWORD3= 'A10     '           / ... and character information
ENDDEF                         / end keyword definition section
This definition section could be written out as a first block on a tape
integrated in a standard FITS header and should be thoroughly commented.
Now, if any of the following headers contained
TELESCOP= 'SEST-15M'          / telescope used
which is one of the allowed keywords in FITS, only then would the program
be prepared within the same header to understand and correctly read any
lines like
SNGLDISH SEST-15M  KEYWORD1=       12.3
SNGLDISH SEST-15M  KEYWORD2=          T
SNGLDISH SEST-15M  KEYWORD3= ' a string '
If the TELESCOP= line had not been present, the SNGLDISH SEST-15M lines
should be ignored. This would allow slightly different implementation of
identical keywords at different telescopes, i.e. everybody may use very
obvious keywords without knowing how exactly they are used somewhere else.
b) Along the lines of adopting IEEE floating point formats for binary
encoding of data, which I very much agree with, let's also adopt a
standard for an extended 8 bit character set. Maybe we could even
agree on how to encode greek, japanese etc. characters. If we want others
to accept a standard proposed by us, we should be open minded about
standards that other commitees have already come up with.

From dishfits-request@fits.CX.NRAO.EDU Fri Oct 27 02:43:04 1989
Received: from NRAO.EDU (cv3.cv.nrao.EDU) by fits.cx.nrao.edu (4.0/SMI-DDN)
	id AA11955; Fri, 27 Oct 89 02:43:02 EDT
Received: from fits.cx.nrao.edu by NRAO.EDU (3.2/DCW-2n    )
	id AA09029; Fri, 27 Oct 89 02:42:51 EDT
Received: by fits.cx.nrao.edu (4.0/SMI-DDN)
	id AA11952; Fri, 27 Oct 89 02:31:57 EDT
Return-Path: <NSSDCA::PSI%RPEPPING::RNORRIS@cvax.CV.NRAO.EDU>
X-Vms-Mail-To: DISHFITS
Message-Id: <891027021112.000045A4071@cvax.CV.NRAO.EDU>
Status: RO
From: "Ray Norris, Australia Telescope, CSIRO, Australia" <NSSDCA::PSI%RPEPPING::RNORRIS@NRAO.EDU>
Sender: dishfits-request@fits.CX.NRAO.EDU
To: dishfits@fits.CX.NRAO.EDU
Subject: Comments from the AT
Date: Fri, 27 Oct 89 02:11:12 EDT
Comments on Single Dish FITS formats and requirements for observing
Ray P. Norris
Australia Telescope National Facility
1 INTRODUCTION
Since the Australia Telescope (AT) can't send anyone to the Greenbank meeting,
this document represents our submission to the meeting. There are three parts
to this submission:
(i) Comments on the structure of a future single-dish FITS standard
(ii) An overview of what we have and where we're going
(iii) An Appendix proposing changes to the FITS standard to make it
more suitable for real-time use.
2 COMMENTS ON FUTURE SINGLE DISH FITS STANDARDS
We would like to change most of our internal data formats to FITS, for
all the usual reasons. To be able to do so, we require that
(i) The FITS standard is extended to make it more suitable for real-time
data logging (see my comments in the Appendix to this submission). In
particular for spectral line use we need to be able to write many spectra
into a file without knowing how many there are to come (see comments by
Norris and by Wilson).
(ii) We need to put in a lot of site-specific information, and so we would
strongly support the concept of hierarchical keywords. The most promising
format for this is option (2) in Preben Grosbol's recent Email (the mix of
Grosbol, Wells, and Hanisch proposals).
(iii) The idea of binary 3-D tables seems excellent, and we would support the
use of IEEE standards for these.
3 THE AT: WHAT WE HAVE AND WHERE WE'RE GOING
The AT consists of (a) the 6km compact array at Culgoora, which is a 6 x 22m
East-West synthesis array with operation planned up to 115 GHz, (b) the
Parkes 64-m antenna which operates up to 43 GHz, (c) the 22-m Mopra (Siding
Spring) antenna which will operate up to 115 GHz, (d) the Long-Baseline
Array which consists of the antennas above plus others used collaboratively
with other Australian groups to produce baselines upto 3000 km.
Here we are concerned only with single-dish observations, and so restrict
attention to the single-dish use of the Parkes and Mopra antennas.
Parkes is used for a broad range of single-dish experiments, including spectral
line studies, polarisation (largely using the Bonn polarimeter), continuum
surveys, and pulsar work. We also hope to install the NRAO multi-beam feed on
Parkes in early 1990. At present most of the software differs radically from
one observing mode to another, and there is little in common either in the
software or in the data format. Mopra is not yet operational, but will become
operational over the next few months, and will be equipped with an 8000-channel
correlator for spectral-line use.
3.1 SINGLE-DISH SPECTRAL LINE OBSERVATIONS
At present spectral line work at Parkes makes use of the SPECTRA observing
program and the 1024-channel correlator. Data output is in a home-grown SPC
format, which many would like to see replaced by FITS. Spectral-line data
reduction is generally done using a 'temporary' program (S) written by Rick
Forster some years ago but which has evolved into the default spectral line
reduction package.
We plan to replace the 1024-channel correlator shortly by a 8000-channel VLSI
correlator (identical to the one for Mopra) and this would be a good
opportunity to overhaul the SPECTRA observing program. It is expected that the
observing software at Parkes and Mopra will be largely identical (except, of
course, for the antenna control software) and so we plan to install a version
of SPECTRA at Mopra.
Whilst many users have come to know and love the S data reduction program, it
is felt that a more sophisticated program would be useful, and so we
are currently flirting with other programs such as SLAP.
3.2 RESPONSE TO QUESTIONS IN ORIGINAL CIRCULAR
In the original circular about the Greenbank meeting, a number of questions
were asked of each observatory. Here are our answers to those questions.
Many of these answers reflect the fact that we are currently undergoing
a transition from an in-house observatory/astronomical institute to a
national facility, and as such we expect to provide a higher level of
support than we have in the past.
(i) What calibration/reduction/analysis software do you have?
We have a variety of observing software, most of which is not suitable for
export. Similarly, most of our data reduction software is site-specific and
not suitable for export. A notable exception is AIPS which is heavily used
for AT, VLA, and VLBI data. We would welcome, and probably participate in,
any initiative to draw on the experience gained with AIPS and other packages
to produce a real user-friendly single-dish spectral-line reduction program.
(ii) How is your software effort organised in-house?
While the bulk of our software effort is our Epping headquarters site,
much of the on-line software has been written either by users or by staff
at the Parkes and Culgoora observatories. Whilst much of this
software has been immensely succesful, there exists no coherent plan for its
development and this is one area that we would like to put some planning into.
(iii)How do you support your users?
 At present we have real-time support for users, plus a limited
amount of data reduction support. This user support will be increased to
a cradle-to-grave philosophy in the future.
(iv) What are your upgrade paths?
Hardware: at present we have VAXes providing real-time computing, and that will
continue for the immediate future. Off-line, our main processor is a Convex
C-220 together with a number of Sun workstations, SPARCstations, etc.
We expect the use of these workstations to increase, and we would like to see
more use made of distributed computing. We are also likely to
acquire a high-end workstation (e.g. a Sun 4/370 with TAAC board) for
algorithm and visualisation development. On the other hand, use of X-windows
means that we may make more use of dumb terminals like X-terminals, Macs,
and PC's in the future.
Software: We have grave concerns over the future of our off-line support,
and believe that community-wide development is the only way in which
sufficient manpower can be brought to bear on the problem of providing
the level of software that we want and expect. Organising and providing
this development is likely to be extremely difficult, however. Meetings
such as this Greenbank meeting are a good first step.
(v) How do you view shared code?
As above - this is the best way forward for data reduction.
(vi) Could single-dish work make better use of AIPS/IRAF/etc.?
It could, but only if manpower is available to include within
these packages the functions that single-dish users need.
APPENDIX: USE OF FITS FOR REAL-TIME DATA LOGGING.
I've already distributed my comments about the need for modifications to the
FITS standard to make it suitable for real-time logging, and I won't
repeat the detailed arguments here. Briefly, however, the argument runs
as follows:
FITS was not designed for on-line use, and isn't, at present, suitable for
on-line use. However, with some small extensions, it could be made suitable for
on-line use. The lack of a good on-line format has resulted in a proliferation
of internal ad hoc formats around the world, with a resulting waste of manpower
and resources in re-inventing various site-specific wheels. Each site then
develops software for transcribing data from its internal format to FITS (more
manpower!) and then observing assistants spend time transcribing tapes. This
will become more of a problem as we move towards using common data reduction
software in several sites.
Perhaps more importantly, as multi-beam feeds and large correlators increase
the volume of data (which could result in peak data rates of 100 kbyte/s in the
near future), the resources needed to transcribe data approach those needed to
take the data in the first place. Transcribing data then becomes a full-time
task rather than a little job to be done before you back up the data, and could
become a significant drain on observatory resources.
I am therefore advocating the extension of FITS to make it suitable for real
time use. To summarise my argument, I contend that (i) to use FITS for on-line
logging makes sense in terms of efficient use of manpower and other resources,
(ii) FITS doesn't let you do this efficiently at present, because FITS requires
you to know in advance how much data you're going to write before you start
writing the file, and in general you don't know this in real-time data logging,
(iii) FITS is fragile if all the tables are put at the end of the data, (iv)
the alternative of writing many small files is cumbersome and causes problems
of data management.
Point (ii) CAN be overcome with present FITS by some form of buffering, but
that is somewhat inefficient and may get impractical as data rates increase. A
better solution is to allow the option of GCOUNT=-1, meaning that you don't
know how many groups you will be writing, and that any FITS reading program
must in this case check each block for the start of a table, or (if scans are
concatenated in a file, which I believe also to be desirable) the start of a
new header. This extension of the FITS standard has no effect whatsoever on the
speed, efficiency, and implementation of present FITS. Those who don't like it
can ignore it, and write out GCOUNT as at present, EXCEPT that the FITS reading
programs must be extended to accomodate the possibility that they may encounter
data with GCOUNT=-1.
Point (iii), the problem of fragility, can be overcome by writing headers and
tables within the body of the file, as well as at the start and end. In the AT
data format, each scan of data is preceded by a FITS-like header and tables,
but all scans are contained within the same file. In this way, the damage
caused by a machine crash or parity error is confined to one scan.
An alternative approach to overcoming fragility has been suggested
by Eric Greisen, who proposes splitting up an observation into
several files - writing a fixed number of records in each tape
file and then starting a new tape file. This approach is similar to mine
except that I prefer to have all the data in one file.
As a general point, I would propose extending the FITS standard to allow many
scans to be recorded in one file - effectively allowing the concatenation of
many FITS files into one file. This approach (plus the others suggested above)
has been used with great effectiveness for the AT data format.
From dishfits-request@fits.CX.NRAO.EDU Fri Oct 27 15:29:34 1989
Received: from NRAO.EDU (cv3.cv.nrao.EDU) by fits.cx.nrao.edu (4.0/SMI-DDN)
	id AA13116; Fri, 27 Oct 89 15:29:31 EDT
Received: from fits.cx.nrao.edu by NRAO.EDU (3.2/DCW-2n    )
	id AA10595; Fri, 27 Oct 89 15:28:40 EDT
Received: by fits.cx.nrao.edu (4.0/SMI-DDN)
	id AA13111; Fri, 27 Oct 89 15:17:13 EDT
Return-Path: <jball@wells.haystack.edu>
Message-Id: <8910271919.AA11805@wells>
Status: RO
From: jball@wells.haystack.edu (John Ball)
Sender: dishfits-request@fits.CX.NRAO.EDU
To: dishfits@nrao.edu
Cc: jball@wells.haystack.edu
Subject: Having FITS
Date: Fri, 27 Oct 89 15:19:40 EDT
                         NEROC Haystack Observatory
                     Westford, Massachusetts  01886  USA
                                 Memorandum
To:            Distribution
From:          John Ball                               1989 October 27
Subject:       Having FITS
        At Haystack, we now have the following three programs to write
FITS tapes from Haystack spectral data:
               1)  FITS reads stashes and writes canonical FITS
        tapes as defined in E.W. Greisen and R.H. Harten, "An
        Extension of FITS for Groups of Small Arrays of Data,"
        Astronomy & Astrophysics Supplement Series 44:371 (June
        1981).  So far as I know, there is no program anywhere
        that can read this FITS format; this FITS program is
        probably useless.
               2)  FITS2 reads stashes and writes CLASSy FITS
        tapes.  CLASS is a radio-astronomical data-reduction
        program produced in France by the Grenoble group,
        released to the public, and being used by Karl Menten and
        John Szczepanski (and others?) at CfA.  CLASS now runs
        only on VAXes, but a version for Unix machines is being
        discussed.
               3)  FITS3 reads any of 11 spectral files (including
        #RTSLD, #BUSLD, #PRTDn) and writes CLASSy FITS tapes.
        FITS for single-dish data should be defined as a file format,
not a tape format.  We can reasonably expect that by the time the
format is implemented, most observatories and most users will be
transferring data over electronic links from one computer to
another.  A tape format might be just a bit-for-bit copy of the
disk format or maybe an observatory-specific format for archiving.
There is no particular reason why an observatory should record or
archive in FITS format, although this should be possible.  The
point is to make FITS available to investigators over an electronic
network and with luggable media.
        I imagine an investigator at his office anywhere calling up
scans into his favorite data-reduction program and reviewing them
on his screen within a few seconds after they were acquired at a
remote telescope.  Alternatively, an investigator at the telescope
should be able to send data to his own computer, run a data-
reduction program on them, view the results at the telescope, and
be sure that the data are safe on his own disk before leaving the
telescope.
        One can imagine arbitrarily complex data formats, and these
may be useful for some specialized purposes.  I urge, however, that
a simplified version of the FITS format should be permissible
within the specifications.  Allow, for example, omission of
parameters that can have default values.  Allow omission of
parameters that can be calculated from other parameters.  Example:
galactic latitude and longitude can be calculated from RA and Dec.
Allow omission of parameters that are just unknown.  Example:
atmospheric attenuation.  (Do you suppose that my program to
calculate atmospheric attenuation from surface meteorology is
better than yours?)  The simplified format should contain raw
data--all the raw data; other parameters can be calculated later.
That is called data reduction.  There is no particular reason why
the reduced data should be kept in FITS format, although this
should be possible.
        Insure that some program somewhere can read the new FITS
format.  How can those of us at observatories test our new FITS-
writing programs if nobody can read their output?  A published
standard is useless and worthless if nobody can use it.
        Make your specifications so clear and so definite that those
of us who have to read and use them, no matter how stupid or
inattentive we are, cannot misunderstand your intentions.  Make
misinterpretation difficult, if not impossible.


From dishfits-request@fits.CX.NRAO.EDU Sat Oct 28 00:20:12 1989
Received: from NRAO.EDU (cv3.cv.nrao.EDU) by fits.cx.nrao.edu (4.0/SMI-DDN)
	id AA13853; Sat, 28 Oct 89 00:20:09 EDT
Received: from fits.cx.nrao.edu by NRAO.EDU (3.2/DCW-2n    )
	id AA11666; Sat, 28 Oct 89 00:20:07 EDT
Received: by fits.cx.nrao.edu (4.0/SMI-DDN)
	id AA13849; Sat, 28 Oct 89 00:05:19 EDT
Return-Path: <POSTMASTER@NRAO.EDU>
X-Vms-Mail-To: DISHFITS AT NRAO
Message-Id: <891028000222.000039BC031@cvax.CV.NRAO.EDU>
Status: RO
From: "GUILLOTE@FRGAG51" <POSTMASTER@NRAO.EDU>
Sender: dishfits-request@fits.CX.NRAO.EDU
To: dishfits@fits.CX.NRAO.EDU
Subject: Too late may be ?
Date: Sat, 28 Oct 89 00:02:22 EDT
Date sent:  28-OCT-1989 04:54:46.17
To:   DISHFITS@NRAO
            Dear FITS designers,
As I won't be able to attend the meeting, I sum up here in a rather
detail way my thoughts about the various propositions which came to
me. I am sorry to be so late. Hope you can get it before leaving,
otherwise Thierry will distribute it.
            Have a nice meeting !
            Stephane Guilloteau
            Institut de Radio Astronomie Millimetrique
            300 Rue de la Piscine
                F-38406 Saint Martin D'Heres
Phone       76 82 49 43
E-Mail      GUILLOTE@FRGAG51.BITNET
----------------------------------------------------------------------
1) Reply to D.Wells proposal for 3-D tables
1.1) Comment on the general philosophy
The 3-D tables idea is quite nice. It is a generalisation of the
"random group" format that Thierry and I were proposing, and is
essentially more flexible. This is both an advantage
and a potential cause of problems.
------------------------------------------------------------------------
| The main difficulty with FITS has always be Reading FITS, not        |
| Writing. This was clearly realized by the original designers         |
| of FITS. It resulted in several compromises such as the              |
| fixed format and rigid alignments of keyword values. Also            |
| in the original papers, it was recommended that                      |
|       "transmitting institutions should write the tapes, whenever    |
|        possible, in a way which will aid the receiving institution"  |
|               (AA Sup p.367, middle of Section 7.)                   |
| and                                                                  |
|       "the user should always try to keep the requirements           |
|       and restrictions of the reading institution in mind"           |
|               (AA Sup p.372, end of section 4.)                      |
------------------------------------------------------------------------
With the 3-D tables, and with the idea of correspondence between
Keywords and Columns ("virtual columns" and "virtual keywords"),
the flexibility is even enhanced and these recommendations should
be further enforced.
One thing we MUST forbid is the use of pointers within the 3-D table.
Pointers would break the correspondence mentionned above.
And of course, the 3-D table should not be exclusive of more
usual random groups.
1.2) Hierarchical keywords
The correspondence principle is somewhat in conflict with the "HKBEGIN - HKEND"
idea for hierarchical keywords. This is because the "virtual
keywords" appear only as argument to keyword TTYPEnn keywords.
Since the "HKBEGIN - HKEND" idea implies an ordering
and (to keep it simple) a continuity also of hierarchical keywords.
The simplest continuity (only) is broken for TTYPEnn arguments.
Only a more subtle form of continuity is still valid.
1.3) Some misunderstandings
In section 3.2, the meaning of "structure" is not clear. Does it mean
"line" (by opposition to "columns") ?
1.4) Some absent issues
As Thierry and I already discussed in one of our E-Mail, an important problem
is that of non-regularly sampled frequency data. I suggest that
the frequency (or wavelength) sampling of the various backends
(if more than one) be handled in a separate 3-D table extension,
placed BEFORE the data table(s).
I do not believe resampling is the general solution for
that (a related comment by Bob Wilson), although it should be used
if possible.
1.5) Implementation constraints
To keep it simple, I) the number of columns should remain relatively
small, and II) the columns should concern only rather "standard"
keywords. A possible way in this direction is to define two
types of 3-D Tables: one for raw data and one for reduced data.
For reduced data, it is possible to handle virtually any observing
project in a single 3-D table, since only few parameters are
actually required.
For raw data, a 3-D table should contain only data for which time
dependant parameters can be assumed constant (e.g. calibration,
pointing corrections, etc...). Otherwise, those parameters
would have to placed in columns, most likely violating
either I) or II) (telescope dependent parameters).
A third level might be considered : archive. In this case anything
could go into the binary table, but probably only the writing
program will be able to read  back such complex data.
1.6) Reading constraints
The FITS reading programs which will support 3-D Tables should
also support the more usual "Random Groups", since it is in fact
a (rigid) subset of the 3-D table capabilities. No extra code
should even be necessary.
2) Preben Grosbol comment on Hierarchical keywords
2.1) I do not fully understand the point
"It MUST be ensured that additions to FITS does not generate
 wrong results even for old readers"
     It seems that the blocking factor extension violate this
 rule? No?
2.2) Array keywords
Continuation cards should be implemented using the last non-comment
character. This is the only way to know a priori that the data
is not complete, so that further reading is required. Simple
string contatenation can then be used, with the final
READ using a (now standard) list-directed format.
All other schemes imply either to read keywords in advance, or
to count the number of arguments.
The full dimension of the array should be specified in the keyword.
Sub-array notations (ARRAY(2..4) or ARRAY(2:4)) are ambiguous or
require a distinct keyword to specify the array size. They should probably
be prohibited.
2.3) Hanisch proposal revisited
As mentionned in section 1.2, it is somewhat conflicting with
the "correspondence" idea. A point to clear up...
3) Ray Norris proposal
3.1) Is the GCOUNT=-1 idea in conflict with the
"It MUST be ensured that additions to FITS does not generate
 wrong results even for old readers"
requirement ?
3.2) Turning back to our previous discussion on FITS for
real-time logging, I still believe this is the true problem has
not been considered. The key is Random Access Buffer Space  (RABS)
(either disk or memory), not only data throughput. If sufficient RABS
is available, FITS file headers can be updated for the current
size before writing to a sequential device.
4) Bob Wilson comments
4.1) multiple spectra :
This is converging with the Ray Norris proposal. It also points out
a possible difficulty in the 3-D tables. If many spectra of
variable length are written in a single 3-D table, extracting
just one may prove costly, since there is no way to predict
the position of the spectrum in the table (other than reading
part of the table). Please NEVER use pointers to circumvent
that small problem: this would lead to lot of confusion (cf 1.1).
4.2) "Core" keywords for reduced data:
I fully agree with this idea. It was behind the proposal 1.5 for
two (or three) levels of 3-D tables.
4.3) "Scan" numbers :
With the advent of multi-beam and multi-backends, this concept
is obsolete. Single-dish data reduction should tend towards
a descriptive approach of the data. The GAG-IRAM program
CLASS is an (uncomplete) attempt in this direction.
5) Pat Wallace comments:
5.1) Pointing corrections
Yes, these are really telescope specific parameters, and nothing
else. They should not even be mentionned for reduced data (see 4.2
and 1.5).
5.2) Source coordinates:
I have in my FITS papers a draft by Bob Hanisch and Don Wells,
entitled "World Coordinate Systems Representations Within the FITS Format".
It describes in a rather complete way conventions to be used for standard
geometries, equinox, etc... What is the status of that draft ?
I only have one very specific request. Drifts in horizontal coordinates
are often used for pointing determination in radio astronomy. That
king of data hardly conveys any useful astronomical content, but
is quite important. Would it be possible to add a -HOR projection
to the list of standard ones, implying that the coordinates
are horizontal offsets from the tangent point at the time of
observation? Or is that too specific?
6) Revised Stobie and Morgan memo
Even though it seems that many people have diverged from this proposal,
some comments might still be useful
6.1) p 22 ASCII versus Binary : Binary only please...
6.2) p 22 TABLE extension versatility for multi-beam or multi-backend
The real issues are:
- when will the processing systems be ready for multi-beam multi-backend?
- how do they handle a multi-something: as a whole or not?
- if not, where should be the separation ?
            - in real-time ?
            - in FITS writing ?
            - in FITS reading ?
            - in processing ?
6.3) p 22 Hierarchical approach :
Lack of space may become a problem if
            HIERARCHY SINGLDSH ANALYSIS Program Structure Variable
is required... Redundant keywords must be suppressed (e.g.
SINGLDSH with ANALYSIS or ANALYSIS with Program).
6.4) p 22 Units
Only SI units please. By the way, why are Degrees used in FITS for angles ?
Radians are the only consistent choice with SI units...
6.5) p 22 Tape layout:
This is against the basic FITS rule. Blocking factors are possible without
specifying the layout.
6.6) P 22 Missing Keywords:
That is right. Missing keywords should not be set to zero. They should
be omitted when writing, and set to Not-A-Number when reading.
6.7) P 22 Keyword repetition:
Not repeating the keyword imply to read and decode all previous extensions
even when only one is wanted. It is a waste of time. The 3-D table
extension more or less suppress this problem if the grouping of data
is done properly.
6.8) p 23 Ill defined keywords:
Those ones were redundant. Suppress them. See also BADCHV,
redundant with the usual BLANK (p 27)
6.9) p 23 Pointing parameters:
Observatory (or even telescope) dependent set. See 5.1
6.10) p 23 Observing parameters and scan numbers:
See 4.3. Each observtory and each analysis program has its own definition
of this number (not even consistent, see IRAM 30-m and CLASS!).
It will be hard to adopt any convention.
6.11) p 23 DESORG and array keywords:
We should play a leading role in array keywords.
6.12) p 24 Map parameters:
Random groups or 3-D binary Tables do that even better, since
no assumption is made on the map sampling.
6.13) p 24 PHASE table:
See 6.11 on array keywords.
7) Three further comments
7.1) What will happen to the DD/MM/YY date format in January 2000 ?
7.2) How should we describe the various velocity reference systems
and related topics in the axis definitions ?
(Frequency, Wavelength, Velocity, Redshift, LSR, Heliocentric,
Geocentric, Radio versus Optical definitions)
7.3) Since implementations of 3-D tables and hierarchical keywords
will be a rather major issue in any case, should we break the
"It MUST be ensured that additions to FITS does not generate
 wrong results even for old readers"
rule by specifying a
SIMPLE =          F     /No "standard" FITS
FORMAT = 'HIERARCHY'    /But correspond the 'HIERARCHY' revision instead
and all further keywords and tape organisation using the full flexibility of
Bob Hanish hierarchical proposal, array notations,
full list-directed input format, 3-D tables, GCOUNT=-1 ?
Wouldn't it be simpler (in terms of programming resources) for most people?
From dishfits-request@fits.CX.NRAO.EDU Thu Oct 26 10:49:38 1989
Received: from NRAO.EDU (cv3.cv.nrao.EDU) by fits.cx.nrao.edu (4.0/SMI-DDN)
	id AA10643; Thu, 26 Oct 89 10:49:36 EDT
Received: from fits.cx.nrao.edu by NRAO.EDU (3.2/DCW-2n    )
	id AA07135; Thu, 26 Oct 89 10:49:27 EDT
Received: by fits.cx.nrao.edu (4.0/SMI-DDN)
	id AA10628; Thu, 26 Oct 89 10:44:08 EDT
Return-Path: <HLISZT@cvax.CV.NRAO.EDU>
X-Vms-Mail-To: DISHFITS
Message-Id: <891026104042.00004B48081@cvax.CV.NRAO.EDU>
Status: RO
From: HLISZT@NRAO.EDU
Sender: dishfits-request@fits.CX.NRAO.EDU
To: dishfits@fits.CX.NRAO.EDU
Subject: Travel plans for visitors to the NRAO
Date: Thu, 26 Oct 89 10:40:42 EDT
Dear People of DishFits;
  At the present time, I can confirm travel plans for the following visitors
to the NRAO.  If you are not on the list but should be, please contact H. Liszt
ASAP via E-mail or by phone at 804-296-0344 (work) or 804-973-3744 (home).  Both
numbers have answering machines which pick up at 3 1/2 - 4 rings.
  W. Baan (Arecibo), R. Barvainis (Haystack), T. Forveille (Grenoble),
  P. Grosbol (ESO), R. Hanisch (StScI), D. Harris (CfA), P. Harrison (Jodrell)
  M. Olberg (Onsala), H. Matthews (JCMT), W. Peters (UA), R. Prestage (JCMT),
  J. Schmidt (MPI), B. Schlesinger (NASA), A. Sievers (MPI), B. Stobie (StScI),
  R. Warmels (ESO).
  Regards to all
    HLiszt@NRAO.EDU
From dishfits-request@fits.CX.NRAO.EDU Sun Oct 29 15:59:50 1989
Received: from NRAO.EDU (cv3.cv.nrao.EDU) by fits.cx.nrao.edu (4.0/SMI-DDN)
	id AA17276; Sun, 29 Oct 89 15:59:49 EST
Received: from fits.cx.nrao.edu by NRAO.EDU (3.2/DCW-2n    )
	id AA15955; Sun, 29 Oct 89 15:59:17 EST
Received: by fits.cx.nrao.edu (4.0/SMI-DDN)
	id AA17273; Sun, 29 Oct 89 15:52:15 EST
Return-Path: <rww@hoh-2.att.com>
>From: hoh-2!rww (R. W. Wilson)
Message-Id: <8910292030.AA22425@hoh-2.>
Status: RO
From: rww@hoh-2.att.com
Sender: dishfits-request@fits.CX.NRAO.EDU
To: arpa!NRAO.edu!DISHFITS@hoh-2.att.com
Date: Sun, 29 Oct 89 15:30:45 EST
>From Bob Wilson, AT&T Bell laboratories
To Dishfits community
Subject: Answers to questions in July 26 letter.
I will not be able to attend the conference, so let me reply to the questions.
1. What calibration/reduction/analysis software do you currently have?
	We use 'comb' (short for combine)  for handling spectral line
	data from the telescope.  It has all of the usual functions
	such as baseline removal, Gaussian fits to lines, etc.  It has
	a well developed macro and do loop system and with a few lines
	of command, all of the data in a given area of the sky from raw
	data files can be put in a 'stacks' file.  At that point,
	images can be made as either space-space maps of any reasonable
	parameter, or space-velocity maps.  These are displayed as
	contour maps and can be written as FITS files.  Images on a regular
	grid can be derived from irregularly sampled data.  Data in the
	'stacks' files are normally retrieved by position (and perhaps
	frequency) rather than stack numbers.
	We usually display the images from comb on our Sun workstations
	using a quick look or movie program.  We use grey scale for single
	images or independent rgb colors when looking at three different
	spectral ranges.  The latter works best with a 24-bit frame buffer
	on one of our Sun's, but can be simulated on the other 8-bit
	workstations using dithering.
	We have IRAF running and use it for most complicated image
	manipulating tasks since we haven't had a working version of
	AIPS since replacing our VAX with the Sun system.  Comb is
	able to make scatter plots and 1-D cuts across an image.
2.  How is your software effort organized in-house?
	Our telescope is on site and a daemon on the Sun file server
	transfers observations over a 9.6Kb/s line to data files on its
	disk as they are taken.  The telescope operation is quite automatic
	and we often monitor progress from our office or home.
	We import data from other observatories (30-meter, SEST, Pom2,
	12-meter, and Nobeyama come to mind).  In each case a special
	purpose program transforms the data to one of comb's formats.
	I am the chief programmer of both OBS and comb.  Marc Pound has
	done a lot with the display programs for the Sun's and John
	Bally has written many of the data translation programs.
3.  How do you support your users?
	We don't have a large stream of visiting observers.  We often
	run the observations for them as that is easier than explaining
	how to do it.  We invite them to come and analyze the incoming data
	after a large enough image is built up.
	Comb is running at several other sites on VAX with BSD style Unix, Sun,
	and HP-UX.
4.  What do you see as logical upgrade paths for:
   A. Hardware
	We work entirely in a UNIX environment (Suns right now) for
	data reduction and we are re-writing OBS for the Lynx operating
	system (a 'real time unix') using an AT buss 80386 machine.
	I think that comb could be ported to almost any Unix machine
	with little effort, but the data presentation programs are
	particular to Sunview. and may have to be re-written (for X-windows?)
	in the future when our hardware changes.
   B. Software
	We put multiple spectrometers in a single 'scan' in our raw data file,
	but treat each as a separate observation in the 'stacks' files.
	I haven't faced up to multiple feeds, since we don't have them yet.
5.  How do you view the prospect of shared code for common use at multiple
observatories?
	I think that the past practice of each observatory writing its
	own code completely is a silly waste of effort.  Does anyone have
	an organized set of routines for pointing, etc.?  I have been trying to
	talk the U.S.Naval Observatory out of some of their code.  Peter
	Stumpf's 'BARVEL' of 10 years ago saved me a lot of effort.
	I would be happy to share any of our data reduction programs with
	anyone wanting to use them.  When the new OBS is completed, we will
	be using it with AST/RO and I would be happy to share it.
6.  Could single-dish work make better use of AIPS/IRAF/other packages not
specifically targeted for it?
	Yes.  We have been doing that for several years and it has been
	very effective.
Robert W. Wilson
rww@hoh-2.att.com
201-888-7120
From PTW%starlink.rutherford.ac.uk@NSFnet-Relay.AC.UK Mon Oct 30 09:59:17 1989
Return-Path: <PTW%starlink.rutherford.ac.uk@NSFnet-Relay.AC.UK>
Received: from NSFnet-Relay.AC.UK ([128.86.8.6]) by fits.cx.nrao.edu (4.0/SMI-DDN)
	id AA18477; Mon, 30 Oct 89 09:58:52 EST
Message-Id: <8910301458.AA18477@fits.cx.nrao.edu>
Received: from starlink.rutherford.ac.uk by NSFnet-Relay.AC.UK
           via Janet with NIFTP  id aa07988; 30 Oct 89 14:17 GMT
Status: RO
From: "P.T.Wallace" <PTW%starlink.rutherford.ac.uk@NSFnet-Relay.AC.UK>
To: DISHFITS-REQUEST <@NSFnet-Relay.AC.UK:DISHFITS-REQUEST@fits.cx.nrao.edu>
Subject:        reply to one of Bob Wilson's points
Date:           Mon, 30 Oct 89  14:19 GMT
>From Pat Wallace, Starlink, UK.
Bob Wilson asks whether there is any pointing software available.
I can offer SLALIB and TPOINT.  Both are written in ANSI Fortran 77,
with a few well-defined and justified extensions.  Both run on
VAX, and SLALIB has been ported to other computers including SUN.
SLALIB is a library of about 140 subroutines mainly concerned
with positional astronomy.  It supports both the old and new
(pre- and post-IAU-1976) systems.
TPOINT is an interactive telescope pointing analysis system.
Further details of these and related products are available from
me or from the Starlink Project.
P.T.Wallace                              19457::PTW
Starlink                                 PTW @ STARLINK.RUTHERFORD.AC.UK
Rutherford Appleton Lab
Chilton, Didcot, Oxon OX11 0QX, UK
-------------------------------------------------------------------------