From fitsbits-request  Tue Mar  1 13:49:23 1994
X-VM-Message-Order:
	(1 3 2 18 19 4 20 6 7 5 21 22 8 9 23
	 26 25 10 29 11 24 27 31 28 32 33 30 12 14 34
	 13 35 15 16 36 37 39 38 40 17 41 42 44 46 45
	 43)
X-VM-Summary-Format: "%n %*%a %-17.17F %-3.3m %2d %4l/%-5c %I\"%s\"\n"
X-VM-Labels: nil
X-VM-VHeader: ("Resent-" "From:" "Sender:" "To:" "Apparently-To:" "Cc:" "Subject:" "Date:") nil
X-VM-Bookmark: 45
Status: RO
X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil]
	["122" "Tue" " 1" "March" "1994" "17:19:25" "GMT" "frankw at sron.ruu.nl" "frankw at sron.ruu.nl" "<1994Mar1.171925.16767 at cc.ruu.nl>" "9" "Q: FTOOLS for HP" "^From:" nil nil "3" "1994030117:19:25" "Q: FTOOLS for HP" (number " " mark "     frankw at sron.ruu.n Mar  1    9/122   " thread-indent "\"Q: FTOOLS for HP\"\n") nil]
	nil)
Received: by fits.cv.nrao.edu (4.1/DDN-DLB/1.5)
	id AA27677; Tue, 1 Mar 94 13:49:23 EST
Return-Path: <news at saips.CV.NRAO.EDU>
Message-Id: <1994Mar1.171925.16767 at cc.ruu.nl>
Organization: ACCU
Path: saips.cv.nrao.edu!hearst.acc.Virginia.EDU!concert!news.duke.edu!MathWorks.Com!europa.eng.gtefsd.com!howland.reston.ans.net!EU.net!sun4nl!news.nic.surfnet.nl!ruu.nl!usenet
Reply-To: frankw at sron.ruu.nl
From: frankw at sron.ruu.nl
Sender: fitsbits-request at fits.CV.NRAO.EDU
To: fitsbits at fits.CV.NRAO.EDU
Subject:  Q: FTOOLS for HP
Date: Tue, 1 Mar 1994 17:19:25 GMT

Hi,

I'm looking for a version of FTOOLS for HP. Does it exist? If so,
where can I get it? 

Thanks,

Frankw at sron.ruu.nl

From fitsbits-request  Tue Mar  1 15:50:01 1994
Status: RO
X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil]
	["3402" "" " 1" "March" "1994" "15:25" "EDT" "BARRY M. SCHLESINGER" "bschlesinger at nssdca.gsfc.nasa.gov" "<1MAR199415254559 at nssdca.gsfc.nasa.gov>" "69" "Re: Unsigned Integers Proposal" "^From:" nil nil "3" "1994030119:25:00" "Unsigned Integers Proposal" (number " " mark "     BARRY M. SCHLESIN Mar  1   69/3402  " thread-indent "\"Re: Unsigned Integers Proposal\"\n") "<2kt28e$kjs at fnnews.fnal.gov>"]
	nil)
Received: by fits.cv.nrao.edu (4.1/DDN-DLB/1.5)
	id AA27825; Tue, 1 Mar 94 15:50:01 EST
Return-Path: <news at saips.CV.NRAO.EDU>
Message-Id: <1MAR199415254559 at nssdca.gsfc.nasa.gov>
Organization: NASA - Goddard Space Flight Center
Path: saips.cv.nrao.edu!hearst.acc.Virginia.EDU!concert!gatech!swrinde!ihnp4.ucsd.edu!ames!news.gsfc.nasa.gov!nssdca.gsfc.nasa.gov!bschlesinger
References: <9402232222.AA08514 at xebec.gsfc.nasa.gov> <2kt28e$kjs at fnnews.fnal.gov>
From: bschlesinger at nssdca.gsfc.nasa.gov (BARRY M. SCHLESINGER)
Sender: fitsbits-request at fits.CV.NRAO.EDU
To: fitsbits at fits.CV.NRAO.EDU
Subject: Re: Unsigned Integers Proposal
Date: 1 Mar 1994 15:25 EDT

In article <2kt28e$kjs at fnnews.fnal.gov>, nicinski at fnal.fnal.gov writes...
>In article <9402232222.AA08514 at xebec.gsfc.nasa.gov>, arots at xebec.gsfc.nasa.gov (Arnold Rots) writes:
>|> Proposal to Add Unsigned Integer Data Types to the FITS Standard
>|> ================================================================
>|> 
>!> [deleted material]
>|> 
>|> As far as integer data types are concerned, FITS only recognizes
>|> unsigned 8-bit and signed (2's complement) 16- and 32-bit integers.
>|> Although there may have been good reasons for this selection in the
>|> past, it is arbitrary, inconsistent, and too restrictive for present
>|> day needs.  I propose that 16- and 32-bit unsigned integers be
>|> recognized, as well.
>|> 
>!> [deleted material]
>!> ... possible solutions:
>|> 
>|> 1. Append the letter "U" to the data type.  A 16-bit unsigned integer
>|> would be represented by "IU", a 32-bit one by "JU".  This has the
>|> advantage that it does not violate the current rules.  However, it is
>|> somewhat distasteful since it looks like another ad hoc kludge.
>|> 
>|> 2. Represent 16-bit unsigned integers by the letter "U", 32-bit ones
>|> by the letter "V".  This choice of letters is reasonably
>|> self-explanatory since it follows the I-J convention.
> 
I think 2. is better.  But the community will have to discuss this 
issue.

>We have taken a different route with unsigned integers under Binary Tables.
>A new logical keyword was created, TSIGNn.  If not present, the current
>"signedness" of the data types is used.  If set to `T', it indicates that
>the field value is signed.  If set to `F', it indicates that the field value
>is unsigned.  Our FITS Table reader will only allow TSIGNn to appear for
>integer data types.
> 
But what about someone else's FITS reader, designed according to the 
definition of BINTABLE in the Cotton, Tody, and Pence proposal?  It 
will ignore the TSIGNn keyword and read the value as a signed integer. 
The whole point of FITS is to create a transport standard that is 
universally recognized.  Local conventions that will not be understood 
by readers elsewhere defeat the purpose of a standard.  Until there is 
some general agreement their use should be limited to those who use 
them.  And they should be designed in a way that will not break 
standard FITS readers.

For any file with the above convention to be FITS conformant, the
binary tables would have to have the value of XTENSION set to other
than BINTABLE, and the new name should be registered with the IAU FITS
Working Group. BINTABLE is already reserved for the Cotton et al.
proposal. Alternatively, the entire file could start with SIMPLE = F,
to indicate that the included binary table does not conform to the
rules for BINTABLE. 

>This type of convention would also work for the PDU.  It's completely backwards
>compatible (as is Arnold's).  We chose a new keyword, rather than a new data type,
>to address PDUs (in case we ever need to do so in the future).  We needed some-
>thing other than TZEROn and TSCALn, because our reader is very generic -- thus
>we cannot afford to perform any data conversions where loss of information is
>possible (we let the user decide how TZEROn and TSCALn are applied).
> 
This proposal is also appropriate for community discussion but should 
not be unilaterally implemented.

				Barry Schlesinger
				NSSDC/NOST FITS Support Office

From fitsbits-request  Tue Mar  1 16:20:12 1994
Status: RO
X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil]
	["580" "Tue" " 1" "March" "1994" "16:20:08" "EST" "William Pence" "pence at tetra.gsfc.nasa.gov" "<9403012120.AA23008 at tetra.Gsfc.NASA.Gov>" "15" "Re: Q: FTOOLS for HP" "^From:" nil nil "3" "1994030121:20:08" "Q: FTOOLS for HP" (number " " mark "     William Pence     Mar  1   15/580   " thread-indent "\"Re: Q: FTOOLS for HP\"\n") nil]
	nil)
Received: by fits.cv.nrao.edu (4.1/DDN-DLB/1.5)
	id AA27860; Tue, 1 Mar 94 16:20:12 EST
Return-Path: <pence at tetra.gsfc.nasa.gov>
Message-Id: <9403012120.AA23008 at tetra.Gsfc.NASA.Gov>
From: pence at tetra.gsfc.nasa.gov (William Pence)
Sender: fitsbits-request at fits.CV.NRAO.EDU
To: fitsbits at fits.CV.NRAO.EDU
Subject: Re: Q: FTOOLS for HP
Date: Tue, 1 Mar 94 16:20:08 EST

> I'm looking for a version of FTOOLS for HP. Does it exist? If so,
> where can I get it? 
> 
> Thanks,
> 
> Frankw at sron.ruu.nl

As far as we know, the standalone version of FTOOLS has not been ported
to an HP.  In principle, the IRAF version of FTOOLS should run on an HP
(assuming IRAF has been ported to your machine), but this has not been
tested.  The ftools distribution .tar files are available from
legacy.gsfc.nasa.gov in the /software/ftools/release directory If you
do get it to run, we would be interested in hearing about it.

-Bill Pence (pence at tetra.gsfc.nasa.gov)

From fitsbits-request  Wed Mar  2 12:52:35 1994
Status: RO
X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil]
	["847" "Wed" " 2" "March" "1994" "16:58:49" "GMT" "Bill Oegerle" "oegerle at stsci.edu" "<1994Mar2.165849.19013 at stsci.edu>" "18" "Re: Unsigned Integers Proposal" "^From:" nil nil "3" "1994030216:58:49" "Unsigned Integers Proposal" (number " " mark "     Bill Oegerle      Mar  2   18/847   " thread-indent "\"Re: Unsigned Integers Proposal\"\n") "<9402281928.AA25016 at fits.cv.nrao.edu>"]
	nil)
Received: by fits.cv.nrao.edu (4.1/DDN-DLB/1.5)
	id AA29656; Wed, 2 Mar 94 12:52:35 EST
Return-Path: <news at saips.CV.NRAO.EDU>
Message-Id: <1994Mar2.165849.19013 at stsci.edu>
Organization: Space Telescope Science Institute, Baltimore MD
Path: saips.cv.nrao.edu!hearst.acc.Virginia.EDU!concert!news-feed-2.peachnet.edu!gatech!swrinde!elroy.jpl.nasa.gov!ncar!noao!stsci!dorado.stsci.edu!oegerle
References: <9402281928.AA25016 at fits.cv.nrao.edu>
Reply-To: oegerle at stsci.edu
From: oegerle at stsci.edu (Bill Oegerle,326,4744)
Sender: fitsbits-request at fits.CV.NRAO.EDU
To: fitsbits at fits.CV.NRAO.EDU
Subject: Re: Unsigned Integers Proposal
Date: Wed, 2 Mar 1994 16:58:49 GMT

In article AA25016 at fits.cv.nrao.edu, forveill at gag.observ-gr.fr (Thierry Forveille) writes:
> I don't find the case for unsigned integers in FITS totally convincing.
> The argument that one should keep the data in as raw a format as possible
> is one that I usually buy, but I would make an exception here since the
> transformation (just an offset, and no scaling) can be inverted exactly,
> to the very last bit included.... (rest deleted)

I agree.  for instance, in IRAF, using the "wfits" task, just use the
parameters:

> wfits file=xxx bscale=1 bzero=32768 bitpix=16 scale=yes

(note that even though you tell IRAF to scale the data, this does not really
happen... you say scale=yes just to get IRAF to use the specified values of
bscale and bzero.  since bscale=1 there is no scaling, just an offset of bzero=32768)

--Bill Oegerle
  STScI

From fitsbits-request  Wed Mar  2 17:17:26 1994
Status: RO
X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil]
	["995" "" " 2" "March" "1994" "21:08:30" "GMT" "Ron Watkins" "ron at argus.lpl.arizona.edu" "<2l2v8e$qca at organpipe.uug.arizona.edu>" "25" "Re: Unsigned Integers Proposal" "^From:" nil nil "3" "1994030221:08:30" "Unsigned Integers Proposal" (number " " mark "     Ron Watkins       Mar  2   25/995   " thread-indent "\"Re: Unsigned Integers Proposal\"\n") "<1994Mar2.165849.19013 at stsci.edu>"]
	nil)
Received: by fits.cv.nrao.edu (4.1/DDN-DLB/1.5)
	id AA01621; Wed, 2 Mar 94 17:17:26 EST
Return-Path: <news at saips.CV.NRAO.EDU>
Message-Id: <2l2v8e$qca at organpipe.uug.arizona.edu>
Organization: Lunar and Planetary Lab, U of AZ
Path: saips.cv.nrao.edu!hearst.acc.Virginia.EDU!concert!gatech!howland.reston.ans.net!europa.eng.gtefsd.com!library.ucla.edu!csulb.edu!nic-nac.CSU.net!news.Cerritos.edu!news.Arizona.EDU!math.arizona.edu!CS.Arizona.EDU!organpipe.uug.arizona.edu!argus.lpl.Arizona.EDU!ron
References: <9402281928.AA25016 at fits.cv.nrao.edu> <1994Mar2.165849.19013 at stsci.edu>
From: ron at argus.lpl.Arizona.EDU (Ron Watkins)
Sender: fitsbits-request at fits.CV.NRAO.EDU
To: fitsbits at fits.CV.NRAO.EDU
Subject: Re: Unsigned Integers Proposal
Date: 2 Mar 1994 21:08:30 GMT

I am unclear on this whole concept of scaling signed values to unsigned values.

Say I have a value in the range 0..65536, Now if these bits are interpreted
as signed values then they are arranged as follows for two's complement:

Unsigned	Signed		Signed+32768
0		0		32768
.		.		.
.		.		.
32767		32767		65535
32768		-32768		0
.		.		.
.		.		.
65536		-1		32767

Now, if I say bscale=1 and bzero=32768, then it will add 32768 to the signed
values. This will goof things up, won't it? What's going on? How can I take
bit patterns that I already have as unsigned and put them into a FITS file
and make the come out again correctly AND at the same time, make it so the
FITS file makes sence of those values?
--
Ron Watkins    [ron at argus.lpl.arizona.edu]    /            /~~~~)     /
931 Gould-Simpson                            /            /____/     /
University of Arizona                       /            /          /
Tucson AZ. 85721 -- (602) 621-8606         (____ unar & / lanetary (____ ab.

From fitsbits-request  Thu Mar  3 09:11:26 1994
Status: RO
X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil]
	["1162" "Thu" " 3" "March" "1994" "09:11:22" "EST" "Arnold Rots" "arots at xebec.gsfc.nasa.gov" "<9403031411.AA02798 at xebec.gsfc.nasa.gov>" "25" "Re: Unsigned Integers Proposal" "^From:" nil nil "3" "1994030314:11:22" "Unsigned Integers Proposal" (number " " mark "     Arnold Rots       Mar  3   25/1162  " thread-indent "\"Re: Unsigned Integers Proposal\"\n") nil]
	nil)
Received: by fits.cv.nrao.edu (4.1/DDN-DLB/1.5)
	id AA03689; Thu, 3 Mar 94 09:11:26 EST
Return-Path: <arots at xebec.gsfc.nasa.gov>
Message-Id: <9403031411.AA02798 at xebec.gsfc.nasa.gov>
From: arots at xebec.gsfc.nasa.gov (Arnold Rots)
Sender: fitsbits-request at fits.CV.NRAO.EDU
To: ron at argus.lpl.arizona.edu
Cc: fitsbits at fits.CV.NRAO.EDU
Subject: Re: Unsigned Integers Proposal
Date: Thu, 3 Mar 94 09:11:22 EST

ron at argus.lpl.Arizona.EDU (Ron Watkins) writes:

> Now, if I say bscale=1 and bzero=32768, then it will add 32768 to the signed
> values. This will goof things up, won't it? What's going on? How can I take
> bit patterns that I already have as unsigned and put them into a FITS file
> and make the come out again correctly AND at the same time, make it so the
> FITS file makes sence of those values?

You are right, this does not work.  In the current situation, you
first have to subtract 32768 from all your numbers, before you deposit
them into the FITS file.

That is why I recommend FITS recognize unsigned integer data types.

Even so, there is one additional complication: Fortran does not know
unsigned integers.  Hence, if your file contains unsigned 16-bit
integers, and you are working in Fortran, you need to put them into
32-bit integers; if you are working with unsigned 32-bit integers, I
guess the only thing you can do officially, is put them into doubles.
If you are working in C, you don't have any of these problems, of
course, but beware: if you are using a Fortran-based FITS reader, you
might still be in for surprises.

  - Arnold Rots

From fitsbits-request  Thu Mar  3 09:46:43 1994
Status: RO
X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil]
	["1489" "" " 3" "March" "1994" "14:22:38" "GMT" "David Robinson" "drtr at mail.ast.cam.ac.uk" "<2l4rre$87m at lyra.csx.cam.ac.uk>" "29" "Re: Unsigned Integers Proposal" "^From:" nil nil "3" "1994030314:22:38" "Unsigned Integers Proposal" (number " " mark "     David Robinson    Mar  3   29/1489  " thread-indent "\"Re: Unsigned Integers Proposal\"\n") "<2l2v8e$qca at organpipe.uug.arizona.edu>"]
	nil)
Received: by fits.cv.nrao.edu (4.1/DDN-DLB/1.5)
	id AA03716; Thu, 3 Mar 94 09:46:43 EST
Return-Path: <news at saips.CV.NRAO.EDU>
Message-Id: <2l4rre$87m at lyra.csx.cam.ac.uk>
Organization: Institute of Astronomy, Cambridge
Path: saips.cv.nrao.edu!hearst.acc.Virginia.EDU!concert!gatech!howland.reston.ans.net!pipex!lyra.csx.cam.ac.uk!pavo.csi.cam.ac.uk!drtr
References: <9402281928.AA25016 at fits.cv.nrao.edu> <1994Mar2.165849.19013 at stsci.edu> <2l2v8e$qca at organpipe.uug.arizona.edu>
From: drtr at mail.ast.cam.ac.uk (David Robinson)
Sender: fitsbits-request at fits.CV.NRAO.EDU
To: fitsbits at fits.CV.NRAO.EDU
Subject: Re: Unsigned Integers Proposal
Date: 3 Mar 1994 14:22:38 GMT

In article <2l2v8e$qca at organpipe.uug.arizona.edu> ron at argus.lpl.Arizona.EDU (Ron Watkins) writes:
>I am unclear on this whole concept of scaling signed values to unsigned
>values.
>...
>Now, if I say bscale=1 and bzero=32768, then it will add 32768 to the signed
>values. This will goof things up, won't it? What's going on? How can I take
>bit patterns that I already have as unsigned and put them into a FITS file
>and make the come out again correctly AND at the same time, make it so the
>FITS file makes sence of those values?

It's very simple. You have data with values x = 0 ... 65535.
So x - 32768 is in the range -32768 ... 32767. Thus the (x-32768) values can be
stored in a BITPIX=16 signed 16-bit integer array. The data has not been
scaled, and has had a constant subtracted from it, therefore bscale = 1 and
bzero = 32768.

Any fits reader will apply the bscale/bzero values to the raw data before
presenting it to the user, thus restoring the original 0 ... 65535 values.

Note that a portable program will *never* be able to simply 'take the bit
patterns that it has as unsigned'; on many computers, the bytes of a native
unsigned int will be in the wrong order compared to the required FITS format.

The existing FITS standard, therefore, can support efficient storage of
unsigned 16-bit integers. Adding a new format would give no extra
functionality, generate incompatibility problems, and require more complex FITS
readers.

 David Robinson. (drtr at mail.ast.cam.ac.uk)

From fitsbits-request  Thu Mar  3 12:00:27 1994
Status: RO
X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil]
	["1561" "Thu" " 3" "March" "1994" "18:00:11" "+0100" "pgrosbol at eso.org" "pgrosbol at eso.org" "<9403031700.AA00687 at ns2.hq.eso.org>" "37" "Re: Unsigned Integers Proposal" "^From:" nil nil "3" "1994030317:00:11" "Unsigned Integers Proposal" (number " " mark "     pgrosbol at eso.org  Mar  3   37/1561  " thread-indent "\"Re: Unsigned Integers Proposal\"\n") nil]
	nil)
Received: by fits.cv.nrao.edu (4.1/DDN-DLB/1.5)
	id AA04040; Thu, 3 Mar 94 12:00:27 EST
Return-Path: <pgrosbol at eso.org>
Message-Id: <9403031700.AA00687 at ns2.hq.eso.org>
From: pgrosbol at eso.org
Sender: fitsbits-request at fits.CV.NRAO.EDU
To: fitsbits at fits.CV.NRAO.EDU
Subject: Re: Unsigned Integers Proposal
Date: Thu, 03 Mar 94 18:00:11 +0100

On the Proposal to Add Unsigned Integer Data Types
       to the FITS Standard by Arnold Rots

The discussion on unsigned 16-bit integers as a FITS
data type started already in the mid 80's when CCD
acquisition systems started to produce 15-16 bit data.

It has been clearly demonstrated in the previous
discussion that it is possible to offset unsigned intergers
and save them as signed values. No information is lost
and the cost is only two interger operations which at
the time of multi MIPS CPUs is insignificant.

If new data types were introduced all FITS reader in the
world would in principle have to be upgraded to conform
to the new standard. This is a dramatic action which can
only be justified if very significant new functionality
is introduced. I believe it was the case when IEEE floating
point formats were added.

It should be noted that the rules for 'Generalized extensions
and blocking factors for FITS' A&A Suppl. 73,p359 state:
  'Only the binary and character coding conventions specified
   in the FITS standards should be used in FITS extensions.'
Thus, one cannot add unsigned intergers to the BINTABLE extension
without changing the full set of FITS data types. Values of
BITPIX=17 or 33 would violate the formular for the size of the
data matrix: nbit = abs(bitpix)*gcount*(pcount+naxis1*...*naxisn)
and are not acceptable.

The proposal would only bring a marginal advantadge (i.e. saving
two integer operations per pixel) but require very significant
code modifications. Thus, I cannot support the proposal.

Preben Grosbol
ESO

From fitsbits-request  Fri Mar  4 15:22:48 1994
Status: RO
X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil]
	["2248" "Fri" " 4" "March" "1994" "15:22:42" "EST" "William Pence" "pence at tetra.gsfc.nasa.gov" "<9403042022.AA13581 at tetra.Gsfc.NASA.Gov>" "38" "Re: Unsigned Integers Proposal" "^From:" nil nil "3" "1994030420:22:42" "Unsigned Integers Proposal" (number " " mark "     William Pence     Mar  4   38/2248  " thread-indent "\"Re: Unsigned Integers Proposal\"\n") nil]
	nil)
Received: by fits.cv.nrao.edu (4.1/DDN-DLB/1.5)
	id AA02503; Fri, 4 Mar 94 15:22:48 EST
Return-Path: <pence at tetra.gsfc.nasa.gov>
Message-Id: <9403042022.AA13581 at tetra.Gsfc.NASA.Gov>
From: pence at tetra.gsfc.nasa.gov (William Pence)
Sender: fitsbits-request at fits.CV.NRAO.EDU
To: fitsbits at fits.CV.NRAO.EDU
Subject: Re: Unsigned Integers Proposal
Date: Fri, 4 Mar 94 15:22:42 EST

Regarding the unsigned integer proposal by A. Rots, my opinion it that
the lack of support for unsigned integers is a serious deficiency of
FITS that needs to be addressed.  This issue is not going to go away no
matter how much we preach that unsigned integer values must be
transformed into scaled integers with the appropriate TSCAL keyword
value.  Many people, as is already evident from the discussion about
this proposal, will simply not follow this FITS requirement and
will go ahead and write the unsigned value to the FITS file.  Some
might add a keyword like SIGNED = F to try to indicate that they broke
the rules, but many won't even do this.  Some people will do this out
of ignorance, not realizing that it is possible to coerce unsigned
integers into a signed integer datatype.  Others may just refuse to do
it because they think this sort of scaling is all rather mindless and
stupid, and they see no point in it.  Still others may have valid concerns
about the impact on efficiency, or on the added complexity of the
software, if they have to scale the unsigned integer data.  So for
whatever reason, people are breaking the FITS rules on this, and I
think the problem will only get worse in the future if more and more
groups start using FITS as their primary data format.  Other competing
data formats, like HDF, do support unsigned integers, so unless FITS can
adapt to the need of users, users will not use FITS.

My preference would be to setup a small technical committee that would
consider all the issues and try to come up with a proposal for
supporting unsigned integers that minimizes the impact on existing FITS
readers.  I've already looked at what it would take to  
add full support for unsigned I*2, I*4, and signed I*1  values in both
primary arrays and in binary tables within my FITSIO subroutine
package and estimate this could be done in a week or 2.  Any
FITS software that uses FITSIO could then transparently read or
write unsigned integers without any change to the application
program, simply by linking to the new FITSIO library.  I think
that the disruption to existing FITS readers is a managable problem,
and the benefits of adding this capability to FITS outweighs the
problems.

-Bill Pence
 HEASARC

From fitsbits-request  Fri Mar  4 15:59:26 1994
Status: RO
X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil]
	["2151" "Fri" " 4" "March" "1994" "21:57:09" "+0100" "Thierry Forveille" "forveill at gag.observ-gr.fr" "<9403042059.AA02558 at fits.cv.nrao.edu>" "40" "Re: Unsigned Integers Proposal" "^From:" nil nil "3" "1994030420:57:09" "Unsigned Integers Proposal" (number " " mark "     Thierry Forveille Mar  4   40/2151  " thread-indent "\"Re: Unsigned Integers Proposal\"\n") "<9403031411.AA02798 at xebec.gsfc.nasa.gov>"]
	nil)
Received: by fits.cv.nrao.edu (4.1/DDN-DLB/1.5)
	id AA02564; Fri, 4 Mar 94 15:59:26 EST
Return-Path: <forveill at gag.observ-gr.fr>
Message-Id: <9403042059.AA02558 at fits.cv.nrao.edu>
Posted-Date: Fri, 4 Mar 1994 21:57:09 +0100
In-Reply-To: <9403031411.AA02798 at xebec.gsfc.nasa.gov>
References: <9403031411.AA02798 at xebec.gsfc.nasa.gov>
From: forveill at gag.observ-gr.fr (Thierry Forveille)
Sender: fitsbits-request at fits.CV.NRAO.EDU
To: arots at xebec.gsfc.nasa.gov (Arnold Rots)
Cc: fitsbits at fits.CV.NRAO.EDU
Subject: Re: Unsigned Integers Proposal
Date: Fri, 4 Mar 1994 21:57:09 +0100

Arnold Rots writes:
 > In the current situation, you
 > first have to subtract 32768 from all your numbers, before you deposit
 > them into the FITS file.
 > 
 > That is why I recommend FITS recognize unsigned integer data types.
 > 
I don't really see what is wrong with subtracting 32768 before writing. The
cost is small compared with the CPU overhead of the physical I/O, and depending
on the computer you are using, you may have to swap the bytes (also a more
costly operation). On RISC architectures, I think instructions for unsigneds 
are actually emulated (16 bits signed instructions too), so CPU efficiency
isn't obtained that way. The additional complexity of the code doesn't seem 
a very serious issue either. Is that rather a philosophical issue of whether 
one is allowed to alter the bits at all?

 > Even so, there is one additional complication: Fortran does not know
 > unsigned integers.  Hence, if your file contains unsigned 16-bit
 > integers, and you are working in Fortran, you need to put them into
 > 32-bit integers; if you are working with unsigned 32-bit integers, I
 > guess the only thing you can do officially, is put them into doubles.
 > If you are working in C, you don't have any of these problems, of
 > course, but beware: if you are using a Fortran-based FITS reader, you
 > might still be in for surprises.
 > 
Most processing packages would in practice put the data in a float,
not in a 32-bit integer. Integers are most frequently used to store what is
really a fixed point real number, at least when there are enough of them
that the size matters. Even photon counts (which are intrinsically integers)
have to be converted to floats for any serious processing or calibration
(rather than just data display). As soon as you need to convert counts
to photons/s/m**2, calibrate the relative response of a spectrograph, or
remove an instrumental background, the data become reals, not integers.
This is another reason why one need not care that the data are stored
as signed with an offset rather than unsigned: what will really be used
is a float or a double.

		Thierry Forveille
		Observatoire de Grenoble

From fitsbits-request  Tue Mar  8 13:19:51 1994
Status: RO
X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil]
	["662" "Tue" " 8" "March" "1994" "05:35:43" "GMT" "BRIAN STAPLES" "brian.staples at digcir.cts.com" "<94030804001821 at digcir.cts.com>" "19" "Need FITS conversion" "^From:" nil nil "3" "1994030805:35:43" "Need FITS conversion" (number " " mark "     BRIAN STAPLES     Mar  8   19/662   " thread-indent "\"Need FITS conversion\"\n") nil]
	nil)
Received: by fits.cv.nrao.edu (4.1/DDN-DLB/1.5)
	id AA09787; Tue, 8 Mar 94 13:19:51 EST
Return-Path: <news at saips.CV.NRAO.EDU>
Message-Id: <94030804001821 at digcir.cts.com>
Organization: Digital Circus BBS (619) 223-5348
Path: saips.cv.nrao.edu!hearst.acc.Virginia.EDU!concert!news.duke.edu!MathWorks.Com!europa.eng.gtefsd.com!howland.reston.ans.net!gatech!udel!news.sprintlink.net!crash!digcir!brian.staples
From: brian.staples at digcir.cts.com (BRIAN STAPLES)
Sender: fitsbits-request at fits.CV.NRAO.EDU
To: fitsbits at fits.CV.NRAO.EDU
Subject: Need FITS conversion
Date: Tue,  8 Mar 1994 05:35:43 GMT

Greetings one and all.

I am looking for a program that will convert GIF, TIF, and/or BMP files
(or other common graphic files) into FITS format. I needed it to run
either in MS WIndows or DOS. A slick program is not what I need, just a
basic conversion program that will allow me to do convert common files
to FITS so I can do some basic image processing (I'm an amateur
astronomer looking to practice).

If anyone out there has any suggestions, recommendations, and sources, I
would truly appreciate it. Many thanks in advance.

Brian Staples

__________________________________________

    bstaples at digcir.cts.com
__________________________________________

From fitsbits-request  Thu Mar 10 09:17:39 1994
Status: RO
X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil]
	["274" "Thu" "10" "March" "1994" "10:15:15" "GMT" "John Boyer" "boyer at rd.eng.bbc.co.uk" "<CMG1tG.5rK at bbc.co.uk>" "10" "please help with imdisp79" "^From:" nil nil "3" "1994031010:15:15" "please help with imdisp79" (number " " mark "     John Boyer        Mar 10   10/274   " thread-indent "\"please help with imdisp79\"\n") nil]
	nil)
Received: by fits.cv.nrao.edu (4.1/DDN-DLB/1.5)
	id AA15584; Thu, 10 Mar 94 09:17:39 EST
Return-Path: <news at saips.CV.NRAO.EDU>
Message-Id: <CMG1tG.5rK at bbc.co.uk>
Organization: British Broadcasting Corporation, UK
Path: saips.cv.nrao.edu!hearst.acc.Virginia.EDU!darwin.sura.net!math.ohio-state.edu!howland.reston.ans.net!pipex!bbc!ant!boyer
From: boyer at rd.eng.bbc.co.uk (John Boyer)
Sender: fitsbits-request at fits.CV.NRAO.EDU
To: fitsbits at fits.CV.NRAO.EDU
Subject: please help with imdisp79
Date: Thu, 10 Mar 1994 10:15:15 GMT

I have imdisp79 and a fits file which i got from stdatu. I have tried to
display this file with imdisp and I get junk. Imdisp says 1024*1024 16
sample bits and the label says 32 sample bits! I am confused can some one
help me please.?

john b
john.boyer at rd.eng.bbc.co.uk

From fitsbits-request  Thu Mar 10 16:52:29 1994
Status: RO
X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil]
	["504" "Thu" "10" "March" "1994" "16:52:24" "EST" "Maureen Conroy" "mo at cfa234.harvard.edu" "<9403102152.AA11370 at cfa234.harvard.edu>" "13" "Re: Unsigned Integers Proposal" "^From:" nil nil "3" "1994031021:52:24" "Unsigned Integers Proposal" (number " " mark "     Maureen Conroy    Mar 10   13/504   " thread-indent "\"Re: Unsigned Integers Proposal\"\n") nil]
	nil)
Received: by fits.cv.nrao.edu (4.1/DDN-DLB/1.5)
	id AA17854; Thu, 10 Mar 94 16:52:29 EST
Return-Path: <mo at cfa234.harvard.edu>
Message-Id: <9403102152.AA11370 at cfa234.harvard.edu>
From: mo at cfa234.harvard.edu (Maureen Conroy)
Sender: fitsbits-request at fits.CV.NRAO.EDU
To: fitsbits at fits.CV.NRAO.EDU
Subject: Re: Unsigned Integers Proposal
Date: Thu, 10 Mar 94 16:52:24 EST

	If there is a well-defined and supported way to
write UNSIGNED integers to FITS files, I would think that the
most important issue is to be sure that the technique is clear
and well-documented.  The NOST FITS office has a FITS users guide
to provide guidelines in using FITS files.  Judging
from the varied responses it would seem to be an excellent example
to cite specifically in the guide, since it seems to be a 
frequently needed feature.  
	Barry - what do you think?

				Maureen Conroy
				SAO

From fitsbits-request  Mon Mar 14 10:01:01 1994
Status: RO
X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil]
	["1823" "" "14" "March" "1994" "09:40" "EDT" "BARRY M. SCHLESINGER" "bschlesinger at nssdca.gsfc.nasa.gov" "<14MAR199409402991 at nssdca.gsfc.nasa.gov>" "35" "Re: Unsigned Integers Proposal" "^From:" nil nil "3" "1994031413:40:00" "Unsigned Integers Proposal" (number " " mark "     BARRY M. SCHLESIN Mar 14   35/1823  " thread-indent "\"Re: Unsigned Integers Proposal\"\n") "<9403102152.AA11370 at cfa234.harvard.edu>"]
	nil)
Received: by fits.cv.nrao.edu (4.1/DDN-DLB/1.5)
	id AA09104; Mon, 14 Mar 94 10:01:01 EST
Return-Path: <news at saips.CV.NRAO.EDU>
Message-Id: <14MAR199409402991 at nssdca.gsfc.nasa.gov>
Organization: NASA - Goddard Space Flight Center
Path: saips.cv.nrao.edu!hearst.acc.Virginia.EDU!caen!saimiri.primate.wisc.edu!nntp.msstate.edu!emory!swrinde!ihnp4.ucsd.edu!ames!news.gsfc.nasa.gov!nssdca.gsfc.nasa.gov!bschlesinger
References: <9403102152.AA11370 at cfa234.harvard.edu>
From: bschlesinger at nssdca.gsfc.nasa.gov (BARRY M. SCHLESINGER)
Sender: fitsbits-request at fits.CV.NRAO.EDU
To: fitsbits at fits.CV.NRAO.EDU
Subject: Re: Unsigned Integers Proposal
Date: 14 Mar 1994 09:40 EDT

In article <9403102152.AA11370 at cfa234.harvard.edu>, mo at cfa234.harvard.edu (Maureen Conroy) writes...
> 
>	If there is a well-defined and supported way to
>write UNSIGNED integers to FITS files, I would think that the
>most important issue is to be sure that the technique is clear
>and well-documented.  The NOST FITS office has a FITS users guide
>to provide guidelines in using FITS files.  Judging
>from the varied responses it would seem to be an excellent example
>to cite specifically in the guide, since it seems to be a 
>frequently needed feature.  
>	Barry - what do you think?
> 

Anything done in the near term has to be done in a way that will not
be misinterpreted by a reasonable FITS reader.  Use of BZERO and
BSCALE satisfies this criterion.  The technique is discussed in
general terms in the first FITS paper.  I will add specific discussion
in the next *full* revision of the User's Guide (version 4.0) about
how unsigned integers can be handled through the use of scaling. 

The TSIGNn proposal does not satisfy this criterion, as FITS files 
using it would be misinterpreted by a FITS reader constructed on the
basis of the current rules of FITS.  Similarly, the proposed new data
types in BINTABLE would not be understood by a current FITS reader.
Such proposed new structures for unsigned integers must be discussed
by the FITS community.  If a consensus should emerge that a new
rule is necessary, then the normal procedures of consideration by
the regional and international FITS committees and demonstration of
its use in data tranport must be followed before it can be approved as
part of FITS.  Present discussion suggests that there is not currently
a consensus as to whether specific provisions for unsigned integers
are desirable.  

				Barry Schlesinger
				NSSDC/NOST FITS Support Office

From fitsbits-request  Mon Mar 14 11:31:26 1994
Status: RO
X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil]
	["7402" "" "14" "March" "1994" "15:49:54" "GMT" "kent at steve.fnal.gov" "kent at steve.fnal.gov" "<2m2132$k78 at fnnews.fnal.gov>" "131" "Yet another unsigned integers proposal" "^From:" nil nil "3" "1994031415:49:54" "Yet another unsigned integers proposal" (number " " mark "     kent at steve.fnal.g Mar 14  131/7402  " thread-indent "\"Yet another unsigned integers proposal\"\n") nil]
	nil)
Received: by fits.cv.nrao.edu (4.1/DDN-DLB/1.5)
	id AA09255; Mon, 14 Mar 94 11:31:26 EST
Return-Path: <news at saips.CV.NRAO.EDU>
Message-Id: <2m2132$k78 at fnnews.fnal.gov>
Organization: FERMILAB, Batavia, IL
Path: saips.cv.nrao.edu!hearst.acc.Virginia.EDU!darwin.sura.net!rsg1.er.usgs.gov!stc06r.CTD.ORNL.GOV!fnnews.fnal.gov!steve.fnal.gov!KENT
Reply-To: kent at steve.fnal.gov
From: kent at steve.fnal.gov
Sender: fitsbits-request at fits.CV.NRAO.EDU
To: fitsbits at fits.CV.NRAO.EDU
Subject: Yet another unsigned integers proposal
Date: 14 Mar 1994 15:49:54 GMT

The Sloan Digital Sky Survey (SDSS) project will generate of order 10 terabytes 
of raw data and 200 gigabytes of reduced data over a period of 5 years.
As computing coordinator for the survey, one of my responsiblities is to
specify the formats for the various types of data that will be collected.
As you can imagine, it is important to do this job properly at the beginning.
These formats will be used not only for internal purposes but also for
the distribution of data to the external community.
Our project was strongly advised to use FITS wherever possible, and we have
been assidiously pursuing that track.

We will use use a combination of basic FITS images, Ascii tables, and binary
tables to store and exchange a variety of raw and processed data.
Our raw data will all be - yes - 16 bit unsigned integers.
We are therefore faced with the same dilemma as Arnold Rots described in
a recent posting, and we have come to the same conclusion - it is very
desirable that the FITS standard provide explicit support for unsigned
data types, and the place to do so is in binary tables.  In spite of the
recent thread arguing that FITS currently handles unsigned types properly
(flip the high order bit and set TSCAL=1, TZERO = 32768),
we have, in fact, found that this is not at all the case.
At present we are faced with writing all of our FITS files with SIMPLE = F
in order that we can adopt some nonstandard conventions to deal with the
inadequacies.  This is highly unsatisfactory.

The first problem we face is one of hard economics - we will be collecting
data at the rate of 7 Mbytes per second, and the current data acquisition
system will not handle the extra compute steps for even a simple bit
flip.  We are not about to spend the several thousand dollars needed
to add the extra compute muscle for this meaningless task.  At present
we will write the raw data as FITS images with the unsupported BITPIX=-16
convention and SIMPLE = F.

The second (and more serious) problem arose when we tried to write
a general purpose reader and writer for FITS images and tables that
translates the data into their most "natural" format - what is the exact
meaning to be applied to data values in the primary array or in tables?
The FITS standard, unfortunately, blurs the distinction between the physical
value that one wants to represent and the FITS table representation of that
value, particularly when scaling is involved.  The FITS standard seems to
imply that a physical value is a pure number plus
physical dimensions, with no connotation of int, float, or signed attached
nor any specific binary representation.  While this connotation might be
adequate for describing data in the primary array (which ought to be IMAGE
data), it is not necessarily appropriate for data in tables (e.g., a table
entry that contains an integer flag to represent a quality code).  Furthermore,
while the standard does define the native data types of a SIMPLE machine in
exquisite detail, it is unclear what, if any, SIMPLE machine representation
is implied when the BSCALE and BZERO (or TSCAL and TZERO for tables) keywords
are present.  The definitions of BSCALE and BZERO (Section 5.2.2.5 of
the NOST standard) imply that, if anything, after scaling, the result is to
be interpreted as a "floating point" number.  The standard does not specify
the exact bit pattern that results after scaling - presumably a FITS reader
must be prepared to deal with an infinite precision floating point number or
with floating point numbers with exponents that are beyond the range of their
IEEE representation. Although it is stated in the draft binary table extension
standard (and has been echoed many times in various supporting documentation
for FITS) that unsigned 16 bit integers may be represented by using signed 16
bit integers and setting BSCALE=1, BZERO=32768, this is not really what the main
standard says. At best, one can say that the use of BSCALE = 1 and BZERO =
32768 results in a floating point number that can, if the user desires, be
represented internally as a 16 bit unsigned integer, but one should not
assume that the scaled FITS data value has that meaning attached.

We will have situations where greater precision of meaning is
needed - i.e., we want to make it clear that a particular item has a
specific type and value.  It would seem that binary tables ought to be able
to do this.  Although unsigned types are not native to most Fortrans,
they are native to ANSI C compilers (and indeed much of the SDSS software
is being written in C, in part because of the support for unsigned data types).
We will also have need for a variable length storage mechanism - a typical
purpose will be to store cutout images of objects detected in the survey, which
vary in size for each object, with the object parameters stored in the main
table.  The heap facility that is proposed for binary tables serves our
needs well but again with one exception - unsigned types are not supported in
heap storage (one cannot even apply TSCAL and TZERO to heap data).

Having ranted and raved for so long, I should at least make a specific
proposal.

IN THE MAIN STANDARD:

1. The definition of "physical value" should
   be clarified (is it is a pure number with dimensions attached?).
   The definition and use of "floating point" should be clarified, since
   it is used to mean both the IEEE format and the ASCII format, which are not
   uniquely related.

2. The exact meaning of scaling with BSCALE and BZERO be clarified.  Does it
   imply that an exact bit pattern should result?

3. In the definitions of BSCALE and BZERO, the statements that values of
   1. and 0. are implied if the keywords are not present should be omitted.
   [The reason for omitting them is that scaling implies a type conversion,
   which may not be the desired effect].

4. In chapter 6 of the main standard, which defines the "official" data types,
   the statement that only these types may appear in both the primary array
   and extension tables should be changed to omit the reference on extension
   tables.  (As it now stands, it would appear that the "X"
   and "L" data types in binary tables already violate this chapter).

IN THE DRAFT BINARY TABLE STANDARD:

1. The goal of binary tables is to provide for the interchange of binary data.
   Each datum has a unique binary representation and a type.  All "commonly
   used" data types are supported, including:

   (L) 8 bit Boolean T or F
   (A) 8 bit ASCII characters
   (I) 16 bit signed integers
   (U) 16 bit unsigned integers
   (J) 32 bit signed integers
   (V) 32 bit unsigned integers
   (E) 32 bit IEEE floating point numbers
   (K) 64 bit signed integers
   (W) 64 bit unsigned integers
   (D) 64 bit IEEE floating point numbers
   (X) A byte string of arbitrary length of undefined type (e.g., a bit mask).

   where the letter in parenthesis is the TFORM value (I have revised Rots'
   suggested designations for unsigned integers - sorry about that).

3. The presence of TSCAL and TZERO implies that the table data value is
   to be scaled and interpreted as a numerical value of data type float with
   no bit pattern implied.  Their use is deprecated if the desired
   result is a unique binary representation and/or type.

Are these proposals sensible? Radical?  Do we have to invent yet another
extension table type?  How "Flexible" is FITS?

Steve Kent

From fitsbits-request  Tue Mar 15 13:22:33 1994
Status: RO
X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil]
	["1541" "" "15" "March" "1994" "17:55:35" "GMT" "Steve Allen" "sla at umbra.ucsc.edu" "<2m4sqn$3ae at darkstar.UCSC.EDU>" "29" "Re: Yet another unsigned integers proposal" "^From:" nil nil "3" "1994031517:55:35" "Yet another unsigned integers proposal" (number " " mark "     Steve Allen       Mar 15   29/1541  " thread-indent "\"Re: Yet another unsigned integers proposal\"\n") "<2m4pup$js7 at paperboy.gsfc.nasa.gov>"]
	nil)
Received: by fits.cv.nrao.edu (4.1/DDN-DLB/1.5)
	id AA12921; Tue, 15 Mar 94 13:22:33 EST
Return-Path: <news at saips.CV.NRAO.EDU>
Message-Id: <2m4sqn$3ae at darkstar.UCSC.EDU>
Organization: UCO/Lick Observatory, Santa Cruz
Path: saips.cv.nrao.edu!hearst.acc.Virginia.EDU!concert!news.duke.edu!MathWorks.Com!europa.eng.gtefsd.com!howland.reston.ans.net!agate!darkstar.UCSC.EDU!umbra.UCSC.EDU!sla
References: <2m2132$k78 at fnnews.fnal.gov> <2m4pup$js7 at paperboy.gsfc.nasa.gov>
From: sla at umbra.UCSC.EDU (Steve Allen)
Sender: fitsbits-request at fits.CV.NRAO.EDU
To: fitsbits at fits.CV.NRAO.EDU
Subject: Re: Yet another unsigned integers proposal
Date: 15 Mar 1994 17:55:35 GMT

In article <2m4pup$js7 at paperboy.gsfc.nasa.gov>,
Robert Hill <bhill at trifle.gsfc.nasa.gov> wrote:
>kent at steve.fnal.gov writes:
>>(flip the high order bit and set TSCAL=1, TZERO = 32768),
>
>Actually, this is not quite right, I think.  You don't `flip' the
>high order bit, you just treat it as a data bit.

No, you simply flip the high order bit and insert BSCALE and BZERO cards
into the header.  That is all.  If your machine can actually add or
subtract faster than it can flip a bit; good, then do that!

This same subject has occurred in the Keck data acquisition system.
The ADCs produce 16-bit unsigned numbers which are all shoved through
a DSP on their way to the permanent storage device.  With the DSP
the processing required to flip the high bit would mean one more
operation on the data while they are in the DSP.  The end result is that
the first processed bits would come out a few microseconds later, but
the overall throughput need not be affected.

The complication for Keck arises further down the software chain because
all successive software, specifically the image preview, must understand
that the data are unsigned.  This requires systems integration between
geographically distinct parallel development teams.

--  --  --  --  --  -+-   --  --  --  --  --  --  --  -+-  --  --  --  --  --
Steve Allen          | "Shameless Morris Enthusiasm"   |   sla at lick.ucsc.edu
UCO/Lick Observatory | Yep, that's me.  6 up for       |   +1 408 459 3046
Santa Cruz, CA 95064 | Orange in Bloom, Sherborne, eh? |   Seabright Morris

From fitsbits-request  Tue Mar 15 13:01:33 1994
Status: RO
X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil]
	["4409" "" "15" "March" "1994" "17:06:33" "GMT" "Robert Hill" "bhill at trifle.gsfc.nasa.gov" "<2m4pup$js7 at paperboy.gsfc.nasa.gov>" "88" "Re: Yet another unsigned integers proposal" "^From:" nil nil "3" "1994031517:06:33" "Yet another unsigned integers proposal" (number " " mark "     Robert Hill       Mar 15   88/4409  " thread-indent "\"Re: Yet another unsigned integers proposal\"\n") "<2m2132$k78 at fnnews.fnal.gov>"]
	nil)
Received: by fits.cv.nrao.edu (4.1/DDN-DLB/1.5)
	id AA12893; Tue, 15 Mar 94 13:01:33 EST
Return-Path: <news at saips.CV.NRAO.EDU>
Message-Id: <2m4pup$js7 at paperboy.gsfc.nasa.gov>
Organization: NASA Goddard Space Flight Center -- InterNetNews site
Path: saips.cv.nrao.edu!hearst.acc.Virginia.EDU!concert!news.duke.edu!MathWorks.Com!europa.eng.gtefsd.com!news.umbc.edu!haven.umd.edu!ames!news.gsfc.nasa.gov!bhill
References: <2m2132$k78 at fnnews.fnal.gov>
From: bhill at trifle.gsfc.nasa.gov (Robert Hill)
Sender: fitsbits-request at fits.CV.NRAO.EDU
To: fitsbits at fits.CV.NRAO.EDU
Subject: Re: Yet another unsigned integers proposal
Date: 15 Mar 1994 17:06:33 GMT

kent at steve.fnal.gov writes:
>The Sloan Digital Sky Survey (SDSS) project will generate of order 10 
>terabytes 
>of raw data and 200 gigabytes of reduced data over a period of 5 years.
[...]
>We will use use a combination of basic FITS images, Ascii tables, and binary
>tables to store and exchange a variety of raw and processed data.
>Our raw data will all be - yes - 16 bit unsigned integers.
>We are therefore faced with the same dilemma as Arnold Rots described in
>a recent posting, and we have come to the same conclusion - it is very
>desirable that the FITS standard provide explicit support for unsigned
>data types, and the place to do so is in binary tables.  In spite of the
>recent thread arguing that FITS currently handles unsigned types properly
>(flip the high order bit and set TSCAL=1, TZERO = 32768),

Actually, this is not quite right, I think.  You don't `flip' the
high order bit, you just treat it as a data bit.  A typical easy
way to do this upon writing is to stuff the two bytes into the
low-order locations within a zeroed-out four-byte word.  Then you
subtract 32768 (in four-byte arithmetic) and write out the
low-order two bytes.  The range 0..65535 is mapped directly into
the range -32768..32767.

[...]
>The first problem we face is one of hard economics - we will be collecting
>data at the rate of 7 Mbytes per second, and the current data acquisition
>system will not handle the extra compute steps for even a simple bit
>flip.

Are you using off-the-shelf hardware interfaces with canned
software, or do you have custom hardware interfaces with at least
the inner acquisition routines written by your project in
assembler?  Your comments made me idly curious about your
situation.  In any case, with a conservative hardware scenario,
e.g. a 25 MHz PC massaging the incoming data, I can see the
conversion costing from 1/2 to 1 second per 7 Mbyte chunk, so yes,
you might have a problem trying to do the conversion at that
stage.

[...]
>The second (and more serious) problem arose when we tried to write
>a general purpose reader and writer for FITS images and tables that
>translates the data into their most "natural" format - what is the exact
>meaning to be applied to data values in the primary array or in tables?
>The FITS standard, unfortunately, blurs the distinction between the physical
>value that one wants to represent and the FITS table representation of that
>value, particularly when scaling is involved.
[...]

Actually, I interpret your question in a less philosophical way.
What you want is for your project internal data files to also be
exportable.  That's fine, we do that too, and many projects these
days seem to take the same approach.  You can assure bit-by-bit
recoverability for internal uses by forcing your own reading
software to do the conversion y=1*x+32768 in integer arithmetic.
The question is whether you really need to assure bit-by-bit
recoverability for the export user -- in effect, whether you 
really need the capability of forcing the user to read the files
in the same way you would.  E.g., for most people, what's wrong
with reading 16 bits into a 32-bit f.p. number with a 24-bit
mantissa?  Eight extra bits!

I'm open minded about this proposal, not convinced either way as
yet, and would like to see more discussion.  Instrumentally
oriented people (I'm one of them) have a tendency to say, `These
are the bits we got, so they are the bits you're going to get.'
If your instrumental format happens to be the same as one of the
FITS representations, you get to have things both ways -- you can
supply a physical calibration in BSCALE and BZERO.  Many FITS
readers allow the user, as an option, to read the FITS data array
into a memory array of exactly the same type with no conversion.
Thus bit-by-bit recoverability is a _de facto_ feature of FITS for
many of us already, for certain data types.  (Leaving aside the
fact that it is an implicit feature of ASCII and of X-type bit
fields in BINTABLEs.)  Should this capability be given some kind
of official status?  If so, then the FITS community is pretty much
obligated to try to support the most common instrumental formats
as FITS data representations.

Bob Hill

--
Robert.S.Hill at gsfc.nasa.gov
Not speaking for any of the following:
Hughes STX Corp., Code 681, NASA/GSFC, Greenbelt, MD 20771
"Don't agree with me until I'm finished talking!" -- Darryl F. Zanuck

From fitsbits-request  Tue Mar 15 18:09:16 1994
Status: RO
X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil]
	["3709" "Tue" "15" "March" "1994" "18:09:11" "EST" "William Pence" "pence at tetra.gsfc.nasa.gov" "<9403152309.AA11220 at tetra.Gsfc.NASA.Gov>" "66" "Re: Unsigned Integers Proposal" "^From:" nil nil "3" "1994031523:09:11" "Unsigned Integers Proposal" (number " " mark "     William Pence     Mar 15   66/3709  " thread-indent "\"Re: Unsigned Integers Proposal\"\n") nil]
	nil)
Received: by fits.cv.nrao.edu (4.1/DDN-DLB/1.5)
	id AA13764; Tue, 15 Mar 94 18:09:16 EST
Return-Path: <pence at tetra.gsfc.nasa.gov>
Message-Id: <9403152309.AA11220 at tetra.Gsfc.NASA.Gov>
From: pence at tetra.gsfc.nasa.gov (William Pence)
Sender: fitsbits-request at fits.CV.NRAO.EDU
To: fitsbits at fits.CV.NRAO.EDU
Cc: pence at tetra.gsfc.nasa.gov
Subject: Re: Unsigned Integers Proposal
Date: Tue, 15 Mar 94 18:09:11 EST

To further support the need for a FITS unsigned datatype in binary
tables, the format for the data coming from the X-ray Timing Explorer
Satellite (XTE) is briefly described bolow.  In most of the previous
discussion of this subject it has been assumed that the entire data
stream consists of unsigned integers, and thus it is simple in
principle to just scale  all the data into signed integers.  But in the
XTE case, both signed and unsigned data values are intertwined in
a complex way within the telemetry stream, so it can be quite
complicated and computationally expensive to identify and scale just
the unsigned telemetry values before writing the data to a FITS binary
table.  It is estimated that XTE will be generating close to 1 Terabyte
of data per year (!), so the computational overhead of having to scale
any unsigned integers before writing them to FITS files is a major
concern.

A somewhat simplified description of the XTE data follows:

The XTE satellite dumps a buffer full of data at periodic time
intervals.  The contents of the buffer depends on the observing mode,
(there are dozens of different possible modes).  The natural way to
store this information in a FITS binary table is to write each buffer
of data, along with a time stamp indicating when the data were
received, into one row of a binary table.  The next buffer of data
would be written to the next row of the table.  All the buffers for a
given observation would be contained in a single table.

Each buffer of data can be futher broken down into individual
components, each of which is logically a different column of the
table.  To give a simple example the, data buffer for a given observing
mode might be 322 bytes long containing 2 scalar 8-bit telemetry
values, followed by a histogram composed of 128 16-bit unsigned values,
followed by another histogram of 64 8-bit unsigned values.  (this is a
hypothetical example, but it is at least representative of what real
XTE data will look like).  So, for this example the most natural lay
out for the binary table would be to have 2 scalar columns followed by
2 vector columns.  The 2 scalar columns  would have a 'B' byte data
type, and the second vector column would have a '64B' datatype.  The
first vector column contains 128 unsigned integers so ideally we would
like to define the data type to be something like '128U' or
'128I*UNSIGNED'.

If FITS directly supports unsigned integers then it is conceptually
simple to convert the XTE data to FITS: as the data comes in, first
identify what observing mode it is in and write the appropriate FITS
header defining all the columns in the table.  Then just copy each
buffer as it comes, byte by byte, into the table, one after another,
until the observation is complete.

But if FITS doesn't directly support unsigned integers, the job is much
more complex.  Before appending each new buffer to the table you have
to figure out which bytes within the buffer represent unsigned
integers, and then scale only them so that they fit within the range of
the FITS signed integer data type.  This process is bound to be rather
computationally expensive. XTE will be generating about 2 Gbytes of
data per day and there is already a lot of concern about how much
computing power will be needed to process this amount of data, so
efficiency is very important.

No final decision has yet been made on how to deal with the XTE
unsigned data values in the FITS files, but it looks rather likely at
this point that the project will simply not have the computer resources
necessary to scale the unsigned values and will have no choice except
to invent a convention for directly encoding the unsigned values in
FITS binary tables.

From fitsbits-request  Tue Mar 15 19:30:00 1994
Status: RO
X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil]
	["2766" "Tue" "15" "March" "1994" "19:29:54" "EST" "William Pence" "pence at tetra.gsfc.nasa.gov" "<9403160029.AA11252 at tetra.Gsfc.NASA.Gov>" "55" "Re: Yet another unsigned integers proposal" "^From:" nil nil "3" "1994031600:29:54" "Yet another unsigned integers proposal" (number " " mark "     William Pence     Mar 15   55/2766  " thread-indent "\"Re: Yet another unsigned integers proposal\"\n") nil]
	nil)
Received: by fits.cv.nrao.edu (4.1/DDN-DLB/1.5)
	id AA13858; Tue, 15 Mar 94 19:30:00 EST
Return-Path: <pence at tetra.gsfc.nasa.gov>
Message-Id: <9403160029.AA11252 at tetra.Gsfc.NASA.Gov>
From: pence at tetra.gsfc.nasa.gov (William Pence)
Sender: fitsbits-request at fits.CV.NRAO.EDU
To: fitsbits at fits.CV.NRAO.EDU
Cc: pence at tetra.gsfc.nasa.gov
Subject: Re: Yet another unsigned integers proposal
Date: Tue, 15 Mar 94 19:29:54 EST

To follow up on Steve Kent's (14 March 1994) proposals for
adding support for additional data types within FITS, I would make
the following specific proposals:

1. Use BITPIX = -16 to represent unsigned 16-bit integers in primary
   arrays and IMAGE extensions.   The SIMPLE keyword would need to
   be set = F until this convention is officially recognized. 
   I think various groups have suggested and used this convention in the
   past, so approving this will simply legalize what is already common practice. 

2. In binary tables, add the following new data types:

    (H)  8 bit signed integers
    (U) 16 bit unsigned integers
    (V) 32 bit unsigned integers

    Existing FITS readers will of course not understand these data types,
    and thus will be unable to parse the binary table at all until 
    they are rewritten because they will not know how many bytes wide
    these columns are.  To ease the transition, the following
    interium convention should be used until such time as these (H,U and V)
    data types are officially approved by the FITS committies.

3.  As an interium solution, the following convention should be used
    to represent the new types of integers in binary tables:

    TFORMn = 'B:SIGNED'     / this is a signed byte column
    TFORMn = 'I:UNSIGNED'   / this is an 16 bit unsigned integer column
    TFORMn = 'J:UNSIGNED'   / this is an 32-bit unsigned integer column

    The use of this convention should be deprecated once the H, U and V
    datatypes are approved.

    Note that this convention is entirely consistent with the current
    binary table definition document which allows any additional characters
    to follow the main datatype character (B, I or J in this case).  The
    use of the ':' is consistent with usage in the 'Substring Array' 
    convention in Appendix C of the Binary Table definition document.  Unlike
    the case with the previous proposal (for using H, U and V to represent
    these datatypes) existing FITS readers will still be able to
    read the data in the table, although the reader may incorrectly
    interpret the values.  FITS files using this convention would need
    to have SIMPLE = F
 
Steve Kent also proposed supporting 64 bit signed and unsigned integers
but I would hesitate to support this simply on the grounds that I don't
know how to generate or read 64 bit integers on machines that don't
support this as a native datatype.  If someone could provide a robust
set of routines for converting between 64-bit integers and other
data types (e.g., 64-bit floats) then I might reconsider this.

These proposals do not allow for 32-bit unsigned or 8-bit signed primary
arrays, but I don't think the current need for these formats is very great.  

-Bill Pence

From fitsbits-request  Tue Mar 15 21:51:03 1994
Status: RO
X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil]
	["2287" "Wed" "16" "March" "1994" "02:50:09" "GMT" "Don Wells" "dwells at nrao.edu" "<DWELLS.94Mar15215010 at fits.cv.nrao.edu>" "48" "Re: Unsigned Integers Proposal" "^From:" nil nil "3" "1994031602:50:09" "Unsigned Integers Proposal" (number " " mark "     Don Wells         Mar 16   48/2287  " thread-indent "\"Re: Unsigned Integers Proposal\"\n") "<9403152309.AA11220 at tetra.Gsfc.NASA.Gov>"]
	nil)
Received: by fits.cv.nrao.edu (4.1/DDN-DLB/1.5)
	id AA13980; Tue, 15 Mar 94 21:51:03 EST
Return-Path: <news at saips.CV.NRAO.EDU>
Message-Id: <DWELLS.94Mar15215010 at fits.cv.nrao.edu>
Organization: nrao
Path: saips.cv.nrao.edu!news.cv.nrao.edu!dwells
References: <9403152309.AA11220 at tetra.Gsfc.NASA.Gov>
From: dwells at nrao.edu (Don Wells)
Sender: fitsbits-request at fits.CV.NRAO.EDU
To: fitsbits at fits.CV.NRAO.EDU
Subject: Re: Unsigned Integers Proposal
Date: Wed, 16 Mar 1994 02:50:09 GMT

In article <9403152309.AA11220 at tetra.Gsfc.NASA.Gov>
    pence at tetra.gsfc.nasa.gov (William Pence) writes:
   ".. XTE will be generating.. 1 Terabyte.. per year .."

The nominal design data rate of the VLBA (my previous RT project) is 
4 GB/day, 1.5 TB/year.  This sort of data rate is indeed large by
historical standards, but it is fairly typical in modern systems.

    ".. [figuring] out which bytes within the buffer represent unsigned
   integers.. [will be].. rather computationally expensive.."

It sounds to me like you have many records with the same structure.  I
suggest that a list structure be computed when the first record of a
given type is encountered, so that the list structure can control
efficient processing of each succeeding record. If the number of types
is small enough, precompile optimized algorithms (e.g. maybe even using
unrolled loops).

    ".. XTE will be generating about 2 Gbytes of data per day.."

2 Gbyte/day is 23 Kbyte/s.  
Even my old PC/XT clone can probably achieve that rate.

    ".. efficiency is very important.. rather likely.. project will
    simply not have the computer resources necessary to scale the
    unsigned values .." 

I doubt that this assertion will stand up to a critical review.

                -*-

Modern RISC processors are sufficiently pipelined that they can
overlap one or more cycles of register operations with the clock
cycles of a memory-to-memory copy.  For high-bandwidth
nearly-I/O-limited systems designers should consider how many times
the bits go into and out of main memory. That is where the action is!
They should also examine the components which are doing the DMA
transfers, how many memory cycles they steal from overlapped CPU
operations, and how many memory-to-memory copies are being done in the
I/O system. One or two register-to-register operations in a hot RISC
engine will pale to insignificance in comparison to several redundant
memory cycles, and such register operations may even be free-of-charge
if pipelining occurs.
--
  Donald C. Wells         Associate Scientist         dwells at nrao.edu
                    http://fits.cv.nrao.edu/~dwells
  National Radio Astronomy Observatory                +1-804-296-0277
  520 Edgemont Road,   Charlottesville, Virginia       22903-2475 USA

From fitsbits-request  Wed Mar 16 11:31:50 1994
Status: RO
X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil]
	["9132" "" "16" "March" "1994" "10:54" "EDT" "BARRY M. SCHLESINGER" "bschlesinger at nssdca.gsfc.nasa.gov" "<16MAR199410544747 at nssdca.gsfc.nasa.gov>" "194" "Re: Yet another unsigned integers proposal" "^From:" nil nil "3" "1994031614:54:00" "Yet another unsigned integers proposal" (number " " mark "     BARRY M. SCHLESIN Mar 16  194/9132  " thread-indent "\"Re: Yet another unsigned integers proposal\"\n") "<2m2132$k78 at fnnews.fnal.gov>"]
	nil)
Received: by fits.cv.nrao.edu (4.1/DDN-DLB/1.5)
	id AA15454; Wed, 16 Mar 94 11:31:50 EST
Return-Path: <news at saips.CV.NRAO.EDU>
Message-Id: <16MAR199410544747 at nssdca.gsfc.nasa.gov>
Organization: NASA - Goddard Space Flight Center
Path: saips.cv.nrao.edu!hearst.acc.Virginia.EDU!darwin.sura.net!convex!cs.utexas.edu!swrinde!elroy.jpl.nasa.gov!ames!newsfeed.gsfc.nasa.gov!nssdca.gsfc.nasa.gov!bschlesinger
References: <2m2132$k78 at fnnews.fnal.gov>
From: bschlesinger at nssdca.gsfc.nasa.gov (BARRY M. SCHLESINGER)
Sender: fitsbits-request at fits.CV.NRAO.EDU
To: fitsbits at fits.CV.NRAO.EDU
Subject: Re: Yet another unsigned integers proposal
Date: 16 Mar 1994 10:54 EDT

In article <2m2132$k78 at fnnews.fnal.gov>, kent at steve.fnal.gov writes...

> ... 
> Our project was strongly advised to use FITS wherever possible...
> Our raw data will all be - yes - 16 bit unsigned integers...

> ... the current data acquisition
> system will not handle the extra compute steps for even a simple bit
> flip.  We are not about to spend the several thousand dollars needed
> to add the extra compute muscle for this meaningless task.  At present
> we will write the raw data as FITS images with the unsupported BITPIX=-16
> convention and SIMPLE = F.

Translating data from a form immediately readable only at the
originating location to a standard form easily read elsewhere is not
meaningless. 

In the original FITS paper, Wells, Greisen and Harten (1981, Astron. 
Astrophys. Suppl. 44,363) write

"The usage of SIMPLE=F may be used for data storage or interchange 
 within an institution or between users with similar interests and 
 computers in those cases in which convenience overwhelms the need for 
 standardization, but in which the outward form of this standard is 
 still convenient to implement.  We do not want to encourage any such 
 departures from the basic FITS standard...."

I interpret the intent to be that if a data set is going to be part of 
a general distributional archive, it should not include data with 
SIMPLE=F; that usage is for individual institutions or a relatively
small common interest group.  Discouraging SIMPLE=F arises from the
fundamental purpose of standardization -- avoiding the need to develop
new software each time a data set is received from a different
location.  While writing with a local format may cut costs at the
originating installation, the requirement for receiving locations to
write their own software will result in a greater total cost to the
community.  

One approach that might be proposed would be to incorporate the
proposed new format in some way in a new extension type, with a
registered name, and distribute a detailed proposal. Those aspects of
the data that do not use unsigned integers can be isolated and read by
a standard FITS reader.  And the new effort can be restricted to
development of routines to read the new extensions, which could be
distributed. But the purpose would not be to develop a special format 
for this one data set, but to start the process of extending the FITS 
standard.

> ... what is the exact
> meaning to be applied to data values in the primary array or in tables?
> The FITS standard seems to
> imply that a physical value is a pure number plus
> physical dimensions, with no connotation of int, float, or signed attached
> nor any specific binary representation.... 
> The definitions of BSCALE and BZERO (Section 5.2.2.5 of
> the NOST standard) imply that, if anything, after scaling, the result is to
> be interpreted as a "floating point" number. 
> The standard does not specify
> the exact bit pattern that results after scaling 

FITS specifies the bit pattern for data transport.  It provides
information to the reader to determine whether the data before scaling
are to be interpreted as integers (countable, discrete values), real
(uncountable, continuous values), or text.  There is no specification
as to the bit pattern the reading software uses to store the data from
the FITS file in its home machine *even before scaling*. That will
depend on such issues as to whether the machine is most significant
bit first or last,  ones- or twos-complement, ASCII or EBCDIC. 

A physical value is a number plus its units.  The units tell whether
the value is inherently integer (number of counts), or real (Hertz).
(The NOST document uses the term "floating point" rather than "real"
because of possible confusion between real vs. imaginary and real vs.
integer.) The interpretation is independent of whether or not the
value is stored in a computer somewhere. 

> - presumably a FITS reader
> must be prepared to deal with an infinite precision floating point number or
> with floating point numbers with exponents that are beyond the range of their
> IEEE representation. 

Using scaling with IEEE floating point is not recommended, precisely 
because of the danger of overflows.  The User's Guide discusses the 
overflow problem specifically.  Use of scaling does not rule out the 
possibility that the physical value is to be interpreted as an 
integer.

Now, one potential problem here is in scaling from one number intended 
as an integer to another intended as an integer using a scale factor 
defined as floating point; round-off may introduce an error.  This 
question needs to be discussed.

> Although it is stated in the draft binary table extension
> standard (and has been echoed many times in various supporting documentation
> for FITS) that unsigned 16 bit integers may be represented by using signed 16
> bit integers and setting BSCALE=1, BZERO=32768, 
> this is not really what the main
> standard says. At best, one can say that the use of BSCALE = 1 and BZERO =
> 32768 results in a floating point number that can, if the user desires, be
> represented internally as a 16 bit unsigned integer, but one should not
> assume that the scaled FITS data value has that meaning attached.

That's because this particular method of representing unsigned 16-bit 
integers is the one that is most widely recommended in the community 
consistent with the rules of FITS.  It is not required; therefore, it 
is not part of the current standard. 

> We will have situations where greater precision of meaning is
> needed - i.e., we want to make it clear that a particular item has a
> specific type and value.  It would seem that binary tables ought to be able
> to do this.  

The TDISPn keywords of binary tables can be used to signify whether a 
table entry is to be considered an integer or a real number, and, if 
real, the number of significant digits.

> ...We will also have need for a variable length storage mechanism - a typical
> purpose will be to store cutout images 
> of objects detected in the survey, which
> vary in size for each object, with the object parameters stored in the main
> table.  The heap facility that is proposed for binary tables serves our
> needs well but again with one exception - unsigned types are not supported in
> heap storage (one cannot even apply TSCAL and TZERO to heap data)....

Not being able to scale heap values is a problem and probably an 
oversight.  One possible proposal would be to change the rule to allow 
the use of TSCALn and TZEROn in fields with P data types, with the 
understanding that the scaling is to be applied to the heap data.

> IN THE MAIN STANDARD:

> 1. The definition of "physical value" should
>    be clarified (is it is a pure number with dimensions attached?).
>    The definition and use of "floating point" should be clarified, since
>    it is used to mean both the IEEE format and the ASCII format, which are not
>    uniquely related.

The definitions should probably be discussed by the technical panel 
that revises the NOST standard to incorporate BINTABLE, IMAGE, and 
blocking. 

> 2. The exact meaning of scaling with BSCALE and BZERO be clarified.  Does it
>    imply that an exact bit pattern should result?

No. See above.  FITS does not specify the bit patterns resulting from 
scaling.

> 3. In the definitions of BSCALE and BZERO, the statements that values of
>    1. and 0. are implied if the keywords are not present should be omitted.
>    [The reason for omitting them is that scaling implies a type conversion,
>    which may not be the desired effect].

Scaling does not imply conversion.

> 4. In chapter 6 of the main standard, which defines the "official" data types,
>    the statement that only these types may appear in both the primary array
>    and extension tables should be changed to omit the reference on extension
>    tables.  (As it now stands, it would appear that the "X"
>    and "L" data types in binary tables already violate this chapter).

The requirement originated in the paper on generalized extensions.  
But the above question about X and L data types makes a good point.

> IN THE DRAFT BINARY TABLE STANDARD:
> 
> 1. ....  All "commonly
>    used" data types are supported, including:
> 
>		...
>   (U) 16 bit unsigned integers
>		...	
>   (V) 32 bit unsigned integers
>		... 
>   (K) 64 bit signed integers
>   (W) 64 bit unsigned integers

The community must reach consensus on whether these new types are 
desirable. 

> 3. The presence of TSCAL and TZERO implies that the table data value is
>    to be scaled and interpreted as a numerical value of data type float with
>    no bit pattern implied.  Their use is deprecated if the desired
>    result is a unique binary representation and/or type.

Scaling does not imply that the data value is to be interpreted as 
real.  FITS does not specify the binary representation of data once it 
leaves the FITS file. It specifies the way the data are to be 
interpreted, but scaling does not imply a floating point 
interpretation of the scaled values. 

				Barry Schlesinger
				NOST FITS Support Office

From fitsbits-request  Wed Mar 16 13:20:39 1994
Status: RO
X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil]
	["717" "" "16" "March" "1994" "16:23:20" "GMT" "Robert Hill" "bhill at trifle.gsfc.nasa.gov" "<2m7bpo$a4v at paperboy.gsfc.nasa.gov>" "22" "Re: Yet another unsigned integers proposal" "^From:" nil nil "3" "1994031616:23:20" "Yet another unsigned integers proposal" (number " " mark "     Robert Hill       Mar 16   22/717   " thread-indent "\"Re: Yet another unsigned integers proposal\"\n") "<2m4sqn$3ae at darkstar.UCSC.EDU>"]
	nil)
Received: by fits.cv.nrao.edu (4.1/DDN-DLB/1.5)
	id AA15581; Wed, 16 Mar 94 13:20:39 EST
Return-Path: <news at saips.CV.NRAO.EDU>
Message-Id: <2m7bpo$a4v at paperboy.gsfc.nasa.gov>
Organization: NASA Goddard Space Flight Center -- InterNetNews site
Path: saips.cv.nrao.edu!hearst.acc.Virginia.EDU!concert!news.duke.edu!MathWorks.Com!europa.eng.gtefsd.com!news.umbc.edu!haven.umd.edu!ames!newsfeed.gsfc.nasa.gov!bhill
References: <2m2132$k78 at fnnews.fnal.gov> <2m4pup$js7 at paperboy.gsfc.nasa.gov> <2m4sqn$3ae at darkstar.UCSC.EDU>
From: bhill at trifle.gsfc.nasa.gov (Robert Hill)
Sender: fitsbits-request at fits.CV.NRAO.EDU
To: fitsbits at fits.CV.NRAO.EDU
Subject: Re: Yet another unsigned integers proposal
Date: 16 Mar 1994 16:23:20 GMT

>>kent at steve.fnal.gov writes:
>>>(flip the high order bit and set TSCAL=1, TZERO = 32768),

>Robert Hill <bhill at trifle.gsfc.nasa.gov> wrote:
>>Actually, this is not quite right, I think.  You don't `flip' the
>>high order bit, you just treat it as a data bit.

>In article <2m4pup$js7 at paperboy.gsfc.nasa.gov>,
sla at umbra.UCSC.EDU (Steve Allen) writes:
>No, you simply flip the high order bit and insert BSCALE and BZERO cards
>into the header.  That is all.

By golly, you're right.  Thanks for the lesson.

Bob Hill

--
Robert.S.Hill at gsfc.nasa.gov
Not speaking for any of the following:
Hughes STX Corp., Code 681, NASA/GSFC, Greenbelt, MD 20771
"Don't agree with me until I'm finished talking!" -- Darryl F. Zanuck

From dwells  Wed Mar 16 14:06:04 1994
Status: RO
X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil]
	["2363" "Wed" "16" "March" "1994" "14:06:03" "EST" "Tom McGlynn" "MCGLYNN at grovx1.gsfc.nasa.gov" "<9403161906.AA15704 at fits.cv.nrao.edu>" "54" "Re: Yet another unsigned integers proposal" "^From:" nil nil "3" "1994031619:06:03" "Yet another unsigned integers proposal" (number " " mark "     Tom McGlynn       Mar 16   54/2363  " thread-indent "\"Re: Yet another unsigned integers proposal\"\n") nil]
	nil)
Received: by fits.cv.nrao.edu (4.1/DDN-DLB/1.5)
	id AA15710; Wed, 16 Mar 94 14:06:04 EST
Return-Path: <dwells>
Message-Id: <9403161906.AA15704 at fits.cv.nrao.edu>
From: Tom McGlynn <MCGLYNN at grovx1.gsfc.nasa.gov>
Sender: fitsbits-request at fits.CV.NRAO.EDU
To: fitsbits
Subject: Re: Yet another unsigned integers proposal
Date: Wed, 16 Mar 94 14:06:03 EST

Forwarded from the fitsbits-request alias. -Don
------- start of forwarded message (RFC 934) -------
Message-Id: <940316134907.29200784 at grovx1.gsfc.nasa.gov>
From: Tom McGlynn <MCGLYNN at grovx1.gsfc.nasa.gov>
To: fitsbits-request at fits.CV.NRAO.EDU
Subject: Re: Yet another unsigned integers proposal
Date: Wed, 16 Mar 1994 13:49:07 -0500 (EST)

With regard to the unsigned int/scaling controversy, I've just done a
few simple benchmarks on my DEC 5000/200.  On my machine (and I would
suspect on many others), the scaling is a simple subtraction, underflows
just wrap around.

****************************************************
	Rate of scaling unsigned I*2's in megabytes/s.

Using IDL:		1.8
Unoptimized C:		1.6
Optimized C (-O4):     10.2

I put a few checks in the loops and generally converted blocks of 10K-20K 
bytes.  I also looked at the assembly code to ensure that the basic
loops didn't get optimized out in the optimized C version.

I don't believe this represents the best one can do if one is really
pressed but already it seems clear that the time required to perform
the conversions will be small compared to do I/O.  I've never gotten
more than 500K bytes/sec I/O rates in real applications.  In practice
we'd be converting smaller chunks which would take longer but this will
be balanced by the fact that not everything is getting converted and
one can probably use tricks to speed things up.

At the 10 Mb/s rate it will take only 200 s per day to handle XTE
data assuming that all data is transformed -- and this is by no
means on state of the art computers.

I think Bill's argument that it's hard to know which data to convert
is specious.  If we know enough to write out the FITS header with
appropriate keyword values to describe which words are going to be
unsigned, then obviously we know which words are going to be unsigned.
Just as FITSIO uses similar information to handle host to IEEE conversions
when writing FITS files, software can use user supplied keywords to
automagically perform the byte scaling.   I would be surprised if we
could not add such a capability within FITSIO itself without violating
the current FITS standards. 

All this puts me in the camp of those who feel that the case for 
incorporating unsigned int's is not demonstrated.

		Tom McGlynn
		Compton Observatory Science Support Center
------- end -------

From fitsbits-request  Wed Mar 16 15:47:54 1994
Status: RO
X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil]
	["2955" "Wed" "16" "March" "1994" "20:46:46" "GMT" "Don Wells" "dwells at nrao.edu" "<DWELLS.94Mar16154646 at fits.cv.nrao.edu>" "58" "Re: Yet another unsigned integers proposal" "^From:" nil nil "3" "1994031620:46:46" "Yet another unsigned integers proposal" (number " " mark "     Don Wells         Mar 16   58/2955  " thread-indent "\"Re: Yet another unsigned integers proposal\"\n") "<2m7bpo$a4v at paperboy.gsfc.nasa.gov>"]
	nil)
Received: by fits.cv.nrao.edu (4.1/DDN-DLB/1.5)
	id AA15829; Wed, 16 Mar 94 15:47:54 EST
Return-Path: <news at saips.CV.NRAO.EDU>
Message-Id: <DWELLS.94Mar16154646 at fits.cv.nrao.edu>
Organization: nrao
Path: saips.cv.nrao.edu!news.cv.nrao.edu!dwells
References: <2m2132$k78 at fnnews.fnal.gov> <2m4pup$js7 at paperboy.gsfc.nasa.gov> <2m4sqn$3ae at darkstar.UCSC.EDU> <2m7bpo$a4v at paperboy.gsfc.nasa.gov>
From: dwells at nrao.edu (Don Wells)
Sender: fitsbits-request at fits.CV.NRAO.EDU
To: fitsbits at fits.CV.NRAO.EDU
Subject: Re: Yet another unsigned integers proposal
Date: Wed, 16 Mar 1994 20:46:46 GMT

In article <2m7bpo$a4v at paperboy.gsfc.nasa.gov>
    bhill at trifle.gsfc.nasa.gov (Robert Hill) writes: ...
 RH> In article <2m4pup$js7 at paperboy.gsfc.nasa.gov>,
         sla at umbra.UCSC.EDU (Steve Allen) writes:
         SA> No, you simply flip the high order bit and insert BSCALE and
         SA> BZERO cards into the header.  That is all.
 RH> By golly, you're right.  Thanks for the lesson.

The bit pattern to be used is 8000-hex. 
In 16-bit unsigned it is 32768 (8*4096). 
In 16-bit 2-s complement it is both +32768 *and* -32768. 
The conversion can be produced by *three* different machine opcodes:

XOR 8000   (exclusive-OR to flip the high bit)
SUB 8000   (subtract +32768)
ADD 8000   (add      -32768)

Tutorial for newcomers: 

In 2-s complement arithmetic you negate a number by complementing the
bit pattern and adding one; the complement of 8000 is 7fff and adding
1 generates a carry and you are back to 8000.  In 2-s complement
arithmetic you throw away carries on addition.  If your unsigned
number is 32769=8001, and you use the SUB (subtract) opcode, the
hardware subtracts 8000=32768 by complementing to get 7fff, adding 1
to get 8000, and then adding that to 8001 to get 0001 (the value to
put in the FITS file), after the carry (10000) is discarded. The ADD
and XOR opcodes produce the same result as the SUB opcode. Another
example is 0001 unsigned, which becomes 8001=-32767 (+32767=7fff,
negation is 8000+1) after conversion.

Note that the exact same set of three operators are used to perform
the reverse conversion from 2-s complement 16-bit signed to 16-bit
unsigned. The two examples cited in the previous paragraph were chosen
to demonstrate this invertibility property of the conversion operation.

Historical note about arithmetic:

The first FITS tape passed in interchange in May 1979 was processed by
a CDC 6400, a 60-bit *ONES-COMPLEMENT* computer. In the 1-s complement
arithmetic system you negate by complementing the bits, and when
adding you add carries from the most significant bit back into the
least significant bit (often called "end-around" carry). During the
early days of FITS a major challenge for people writing and reading
FITS files on various machines was to understand how to interchange
between 1-s complement and 2-s complement architectures. Mercifully,
almost all of the 1-s complement machines have gone off to computer
Valhalla.  FITS itself was carefully defined to use 2-s complement.

I am intrigued by the fact that each number system has a number whose
negation is arithmetically equal to itself. In the 1-s complement
system the number is zero ((+0)==(-0) in C-speak), while in 2-s
complement it is 32768 ((+32768)==(-32768)).
--
  Donald C. Wells         Associate Scientist         dwells at nrao.edu
                    http://fits.cv.nrao.edu/~dwells
  National Radio Astronomy Observatory                +1-804-296-0277
  520 Edgemont Road,   Charlottesville, Virginia       22903-2475 USA

From fitsbits-request  Wed Mar 16 16:30:55 1994
Status: RO
X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil]
	["1172" "Wed" "16" "March" "1994" "16:30:40" "EST" "William Pence" "pence at tetra.gsfc.nasa.gov" "<9403162130.AA00273 at tetra.Gsfc.NASA.Gov>" "22" "TSCAL  and TZEROn  with P datatypes" "^From:" nil nil "3" "1994031621:30:40" "TSCAL  and TZEROn  with P datatypes" (number " " mark "     William Pence     Mar 16   22/1172  " thread-indent "\"TSCAL  and TZEROn  with P datatypes\"\n") nil]
	nil)
Received: by fits.cv.nrao.edu (4.1/DDN-DLB/1.5)
	id AA15915; Wed, 16 Mar 94 16:30:55 EST
Return-Path: <pence at tetra.gsfc.nasa.gov>
Message-Id: <9403162130.AA00273 at tetra.Gsfc.NASA.Gov>
From: pence at tetra.gsfc.nasa.gov (William Pence)
Sender: fitsbits-request at fits.CV.NRAO.EDU
To: fitsbits at fits.CV.NRAO.EDU
Subject: TSCAL  and TZEROn  with P datatypes
Date: Wed, 16 Mar 94 16:30:40 EST

Barry Schlesinger wrote:

>> table.  The heap facility that is proposed for binary tables serves our
>> needs well but again with one exception - unsigned types are not supported in
>> heap storage (one cannot even apply TSCAL and TZERO to heap data)....
>
>Not being able to scale heap values is a problem and probably an 
>oversight.  One possible proposal would be to change the rule to allow 
>the use of TSCALn and TZEROn in fields with P data types, with the 
>understanding that the scaling is to be applied to the heap data.

I also think this was just an oversight.  The main text of the binary
table proposal is correct in stating that the use of the TSCALn keyword
is not defined for the P format fields, because the use of the P format
is also not defined in the main text.  The use of TSCALn and TZEROn to
refer to the heap data in P type fields probably should have been
discussed in appendix A, however, where the 'variable length array'
facility is described in detail.  Since the convention described in
Appendix A is "still undergoing trials" and has not been offically
voted on, there should be little problem in correcting this oversight.

-Bill Pence

From fitsbits-request  Wed Mar 16 21:52:36 1994
Status: RO
X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil]
	["1078" "Wed" "16" "March" "1994" "21:52" "EST" "Wayne H. Warren, Jr." "W3WHW at gibbs.gsfc.nasa.gov" "<9403170252.AA16169 at fits.cv.nrao.edu>" "26" "1's Complement Arithmetic" "^From:" nil nil "3" "1994031702:52:00" "1's Complement Arithmetic" (number " " mark "     Wayne H. Warren,  Mar 16   26/1078  " thread-indent "\"1's Complement Arithmetic\"\n") nil]
	nil)
Received: by fits.cv.nrao.edu (4.1/DDN-DLB/1.5)
	id AA16175; Wed, 16 Mar 94 21:52:36 EST
Return-Path: <W3WHW at GIBBS.GSFC.NASA.GOV>
Message-Id: <9403170252.AA16169 at fits.cv.nrao.edu>
From: "Wayne H. Warren Jr. 474-0814"       <W3WHW at GIBBS.GSFC.NASA.GOV>
	        (Code 681/GSFC)
Sender: fitsbits-request at fits.CV.NRAO.EDU
To: FITSBITS                            <fitsbits at FITS.CV.NRAO.EDU>
Subject: 1's Complement Arithmetic
Date:    Wed, 16 Mar 94 21:52 EST

On 16 Mar at 15:47:54EST, dwells at NRAO.EDU(Don Wells) wrote:
> . . .   Mercifully, almost all of the 1-s complement machines have
>gone off to computer Valhalla.  FITS itself was carefully defined to
>use 2-s complement.

>I am intrigued by the fact that each number system has a number whose
>negation is arithmetically equal to itself. In the 1-s complement
>system the number is zero ((+0)==(-0) in C-speak), while in 2-s
>complement it is 32768 ((+32768)==(-32768)).

Not disagreeing with Don's statement, it certainly was convenient for
an observational astronomer working with declinations in sexigessimal
form that the 1's complement CDC machines distinguished between +0 and
-0 and one didn't need to treat the signs independently.  I can't
recount how many times I've seen errors in extinction coefficients and
other results because of the -0 bug in someone's program.

Wayne H. Warren Jr.
Hughes STX Corporation
Laboratory for Astronomy and Solar Physics
Code 681
NASA GSFC
Greenbelt  MD 20771-0001
(301)286-5419 (GSFC); (301)441-4086 (HSTX)
w3whw at gibbs.gsfc.nasa.gov

From fitsbits-request  Fri Mar 18 11:18:43 1994
Status: RO
X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil]
	["3932" "" "18" "March" "1994" "15:50:01" "GMT" "Tom Nicinski" "nicinski at fncasefnal.fnal.gov" "<2mcij9$p9s at fnnews.fnal.gov>" "78" "Re: Yet another unsigned integers proposal" "^From:" nil nil "3" "1994031815:50:01" "Yet another unsigned integers proposal" (number " " mark "     Tom Nicinski      Mar 18   78/3932  " thread-indent "\"Re: Yet another unsigned integers proposal\"\n") "<9403160029.AA11252 at tetra.Gsfc.NASA.Gov>"]
	nil)
Received: by fits.cv.nrao.edu (4.1/DDN-DLB/1.5)
	id AA19703; Fri, 18 Mar 94 11:18:43 EST
Return-Path: <news at saips.CV.NRAO.EDU>
Message-Id: <2mcij9$p9s at fnnews.fnal.gov>
Organization: FERMILAB, Batavia, IL
Path: saips.cv.nrao.edu!hearst.acc.Virginia.EDU!darwin.sura.net!rsg1.er.usgs.gov!stc06r.CTD.ORNL.GOV!fnnews.fnal.gov!fncase!nicinski
References: <9403160029.AA11252 at tetra.Gsfc.NASA.Gov>
Reply-To: nicinski at fnal.fnal.gov
From: nicinski at fncasefnal.fnal.gov (Tom Nicinski)
Sender: fitsbits-request at fits.CV.NRAO.EDU
To: fitsbits at fits.CV.NRAO.EDU
Subject: Re: Yet another unsigned integers proposal
Date: 18 Mar 1994 15:50:01 GMT

I am also a big supporter of a richer environment -- in particular, supporting
additional primitive data types that are quite "natural" and common.  In this,
thread, that means unsigned integers and signed 8-bit integers.  Thus, I agree
with Steve Kent's proposal.  I also have some comments on William Pence's post.

pence at tetra.gsfc.nasa.gov (William Pence) writes:

[deleted ...]

|> 1. Use BITPIX = -16 to represent unsigned 16-bit integers in primary
|>    arrays and IMAGE extensions.   The SIMPLE keyword would need to
|>    be set = F until this convention is officially recognized. 
|>    I think various groups have suggested and used this convention in the
|>    past, so approving this will simply legalize what is already common practice. 

	Although I do not have any suggestions, this convention will only confuse
	the standard.  There are two problems: 1) negative BITPIX values indicate
	floating point values and 2) how do you handle 32-bit or 64-bit or n-bit
	integers.

|> 2. In binary tables, add the following new data types:
|> 
|>     (H)  8 bit signed integers

	I chose W for 8-bit signed integers for my Binary Table reader.

[deleted ...]

|> 3.  As an interium solution, the following convention should be used
|>     to represent the new types of integers in binary tables:
|> 
|>     TFORMn = 'B:SIGNED'     / this is a signed byte column
|>     TFORMn = 'I:UNSIGNED'   / this is an 16 bit unsigned integer column
|>     TFORMn = 'J:UNSIGNED'   / this is an 32-bit unsigned integer column
!> 
|>     The use of this convention should be deprecated once the H, U and V
|>     datatypes are approved.

	I have an aversion to interim proposals, since they kind of become
	standard.  If you're willing to use SIMPLE=F, then using proposal
	point 2. (above) might as well be used from the start.

[deleted ...]

|> Steve Kent also proposed supporting 64 bit signed and unsigned integers
|> but I would hesitate to support this simply on the grounds that I don't
|> know how to generate or read 64 bit integers on machines that don't
|> support this as a native datatype.  If someone could provide a robust
|> set of routines for converting between 64-bit integers and other
|> data types (e.g., 64-bit floats) then I might reconsider this.
|> 
|> These proposals do not allow for 32-bit unsigned or 8-bit signed primary
|> arrays, but I don't think the current need for these formats is very great.

	These last two paragraphs highlight what I see as a severe problem with
	the how people think about FITS.  Rather than trying to provide a richer
	and consistent environment, peoples' own needs are applied to the stan-
	dard.  Why don't people forsee the need for 8-bit signed primary arrays?
	Maybe your application doesn't need them, but someone else's might.

	Why argue against 64-bit integers?  If your platform doesn't like them,
	that should not preclude their inclusion in the standard.  You might
	argue then that the Binary Table complex numbers should be removed.
	I never use FORTRAN and none of the ANSI C compilers and libraries I
	use recognize complex numbers as a primitive data type.  Now, here's
	a significant part of the computing world that cannot handle complex
	numbers naturally; so, why are they in the proposed standard?

	I'm new to the FITS world, but it certainly seems that the 'F' in
	Flexible certainly does not apply.  This standard is not evolving
	to take into account real world problems and needs.

+--- . .' ----+--------------------------------+----------------------------+
|   /| | |  | | Tom Nicinski / MS 234          | nicinski at fnal.fnal.gov     |
|    +-+-+\ | | Fermi National Accelerator Lab |                         _  |
| .  | | | \| | P.O. Box 500                   |                        [o] |
| `--' | |  | | Batavia, IL 60510-0500         |                        /|\ |
+---- -' -----+--------------------------------+----------------------------+

From fitsbits-request  Fri Mar 18 11:38:31 1994
Status: RO
X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil]
	["6655" "Fri" "18" "March" "1994" "11:38:13" "EST" "Arnold Rots" "arots at xebec.gsfc.nasa.gov" "<9403181638.AA11251 at xebec.gsfc.nasa.gov>" "129" "Re: Unsigned Integers Proposal" "^From:" nil nil "3" "1994031816:38:13" "Unsigned Integers Proposal" (number " " mark "     Arnold Rots       Mar 18  129/6655  " thread-indent "\"Re: Unsigned Integers Proposal\"\n") nil]
	nil)
Received: by fits.cv.nrao.edu (4.1/DDN-DLB/1.5)
	id AA19750; Fri, 18 Mar 94 11:38:31 EST
Return-Path: <arots at xebec.gsfc.nasa.gov>
Message-Id: <9403181638.AA11251 at xebec.gsfc.nasa.gov>
From: arots at xebec.gsfc.nasa.gov (Arnold Rots)
Sender: fitsbits-request at fits.CV.NRAO.EDU
To: fitsbits at fits.CV.NRAO.EDU
Cc: arots at xebec.gsfc.nasa.gov
Subject: Re: Unsigned Integers Proposal
Date: Fri, 18 Mar 94 11:38:13 EST

I have watched the debate over the issue of introducing unsigned
integers into the FITS Binary Tables closely and my conclusion (not
surprising, since I started this round in the discussion) is: we
should do it and we'd better do it quickly.

The Binary Table standard contains a fairly large and fairly complete
set of data types.  The criterion for defining this set cannot have
been to arrive at a minimum set.  Considering the fact that it
contains complex and double complex, the approach must have been to
include all meaningful data types commonly in use among astronomers.
I would submit that unsigned integers ought to be included in this
collection, simply on the basis of generality.

Frankly, I think, there is too much parochial thinking going on.  I
have no intention to pick on anyone in particular, but I understand
very well where Don Wells is coming from: in radio astronomy floats,
doubles, complex, signed integers and scaled integers have
traditionally dominated the zoo of data types.  Among the latter
especially the 16 bit variety, as illustrated by Don's remark:

> I am intrigued by the fact that each number system has a number whose
> negation is arithmetically equal to itself. In the 1-s complement
> system the number is zero ((+0)==(-0) in C-speak), while in 2-s
> complement it is 32768 ((+32768)==(-32768)).

Note that this implicitly assumes that all integers are 16-bit signed
integers.  I am sure that Don did not mean it that way, I but mention
it to illustrate the implicit assumptions we are all making.  My point
is that in the rest (i.e., non-radio) of astronomy, complex (for
instance) does not play such a big role, but unsigned integers do.

Again, if we *can* store original values in FITS tables, I feel it
*should* be done.  Every extra line of code increases complexity and
chances for errors; this is of specific concern to those among us who
work on software that will produce basic archives: the number of
operations should be kept to an absolute minimum, on the one hand,
while the archived product should provide as good data access as
possible.  Unsigned integers would help tremendously in that regard.

All I am asking for is that unsigned integers be recognized in Binary
Tables only.  The Binary Tables proposal has only just been approved,
so, since unsigned integers are unavoidably coming, as Bill Pence
pointed out, the sooner they are recognized, the easier the phasing in
will be.

On to the XTE case.  Don Wells and Tom McGlynn have provided us with
some quick calculations that all of us (I hope) are capable of
carrying out but that, in their simplicity, completely miss the point.
The problem is not in the actual scaling itself, though I might point
out that we will decline Don's offer of his PC/XT clone, since we hope
to be able to process a day's worth of telemetry in considerably less
than 24 hours.  Figuring out when, where, and how to scale, involves at
least the same amount of processing.

More important, though, is the added complexity.  Of course, it can be
determined what needs to be scaled by what.  And, of course, it can be
table driven.  But we are talking about a significant increase in code
complexity.  XTE will generate something of the order of a TByte a
year in FITS files, containing data in something of the order of a
thousand data modes (Bill Pence was a little optimistic with his
"dozens").  In order to keep this from turning into a configuration
control and software maintenance nightmare, we have neatly
encapsulated the various functions: the data management subsystem (in
C++) can provide any client (not just a FITS writer) with data values;
the FITS writer (in C) takes the values and dumps them into FITS
Binary Tables.  We don't want to force the former to comply with FITS
standards (for obvious reasons), while we also want to keep the latter
as simple as possible.  I cannot over-stress the importance of this
point: this is crucial software that creates *the* data archive for
this mission; every increase in its complexity increases the risk of
errors, in implementation and maintenance. Unnecessary capabilities
and sophistication is just not acceptable for this type of software.
The FITS writer currently has no direct knowledge of what it is
writing (at least not while it happens) because there is no need for
that capability; strict encapsulation and design simplicity have made
this piece of software highly reliable - I would like to keep it that
way.

For Binary Tables, I would propose the following data types.  New ones
are indicated with an asterisk (*).  I am not quite sure whether
64-bit integers are warranted at the moment, but considering that we'd
better do this right for the next 10 years, or so, I have included
them.  I hope nobody will mistake "H" for Hollerith.

    (L)  8 bit logical
    (X)  1 bit item
    (A)  8 bit ASCII character
    (E) 32 bit float
    (D) 64 bit double float
    (C) 64 bit complex float (2 times 32 bits)
    (M)128 bit complex float (2 times 64 bits)
  * (H)  8 bit signed integer
    (I) 16 bit signed integer
    (J) 32 bit signed integer
  * (K) 64 bit signed integer
    (B)  8 bit unsigned integer
  * (U) 16 bit unsigned integer
  * (V) 32 bit unsigned integer
  * (W) 64 bit unsigned integer

In addition, of course, there is the P.

There is one more issue that I would like to address.  Both Bill Pence
and Barry Schlesinger have hinted at how unsigned integers could be
used in FITS files until such a time as when we have an official
standard.  I am very hesitant to use devices like:
    TFORMn  = 'I:UNSIGNED'   / this is an 16 bit unsigned integer column
or:
    SIMPLE  =              F / This is my type of FITS file
or:
    XTENSION= 'MYBINTABLE'   / A BINTABLE with unsigned integers
as interim measures.  If a standard got established at any point
during the XTE mission, we would either end up with an archive that
contains different types of files, or face the task of reprocessing
all previous data.  Neither is palatable.  The point is that such
devices are quite acceptable for limited amounts of, more or less,
volatile data.  For definitive archives (which, one hopes, would also
outlive software systems that are currently in use) other
considerations come into play.

I sincerely hope that the community will move forward with this
expeditiously.  As Bill Pence remarked, this problem is not going
away; therefore, the sooner we deal with it, the better.  And to
require everybody to deal with the lack of unsigned integers in the
future, because "we have always managed to do without" in the past,
is, in my view, a dangerous position.

  - Arnold Rots

From fitsbits-request  Fri Mar 18 12:53:02 1994
Status: RO
X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil]
	["7632" "Fri" "18" "March" "1994" "18:57:27" "+0100" "Lucio Chiappetti" "lucio at ifctr.mi.cnr.it" "<Pine.3.89.9403181728.E22808-0100000 at poseidon.ifctr.mi.cnr.it>" "153" "Re: Unsigned Integers Proposal" "^Resent-Date:" nil nil "3" "1994031817:57:27" "Unsigned Integers Proposal" (number " " mark "     Lucio Chiappetti  Mar 18  153/7632  " thread-indent "\"Re: Unsigned Integers Proposal\"\n") "<9403181638.AA11251 at xebec.gsfc.nasa.gov>"]
	nil)
Received: by fits.cv.nrao.edu (4.1/DDN-DLB/1.5)
	id AA19849; Fri, 18 Mar 94 12:53:02 EST
Return-Path: <lucio at ifctr.mi.cnr.it>
Organization: Istituto di Fisica Cosmica e Tecnologie Relative
Reply-To: Lucio Chiappetti <lucio at ifctr.mi.cnr.it>
In-Reply-To: <9403181638.AA11251 at xebec.gsfc.nasa.gov>
Message-Id: <Pine.3.89.9403181728.E22808-0100000 at poseidon.ifctr.mi.cnr.it>
Mime-Version: 1.0
Content-Type: TEXT/PLAIN; CHARSET=US-ASCII
Resent-Date: Fri, 18 Mar 1994 18:57:27 +0100 (MET)
Resent-From: fitsbits-request
Resent-Message-Id: <9403181753.AA19849 at fits.cv.nrao.edu>
Resent-Sender: Lucio Chiappetti <lucio at ifctr.mi.cnr.it>
From: Lucio Chiappetti <lucio at ifctr.mi.cnr.it>
Sender: fitsbits-request at fits.CV.NRAO.EDU
To: fitsbits at fits.CV.NRAO.EDU
Subject: Re: Unsigned Integers Proposal
Date: Fri, 18 Mar 1994 18:57:27 +0100 (MET)

Here are my 0.02 currency units (Ukrainian coupons ?) about the 
"unsigned integers in FITS" controversy, which I have been following
for a while. I apologize if in the posting below I do not acknowledge
all previous contributors by name, but I am quoting off my memory.

---------

I have always been liking Ockham's razor, in this particular case
"data types non sunt multiplicanda praeter necessitatem" : the number
of data types should be kept to a minimum.

So just on this ground, I have looked favourably at the various
contributors which argued very convincingly that unsigned integers
can be handled with the BSCALE/BZERO convention without an unreasonable
CPU load and therefore remain of the opinion that one should live
with a LIMITED number of data types.

------------

The usage of a minimal subset could also be helpful to developers in
different languages. Personally I use only Fortran, and I would be
annoyed to deal with unsigned integers which are not supported by
the language (see however below for bytes et al.). One contributor
noted that C programmers may feel the same about the COMPLEX data type
(actually I believe I've used it in Fortran only once, and that was
not serious work, a Mandelbrot set game).

In my opinion the really useful set (which does not mean I propose to
delete what is already in the standard beyond it) is :

  - 8-bit characters                               B
          or bytes (unsigned ....)                 C
  - 16-bit signed integers                         B
  - 32-bit signed integers                         C
  - 32-bit IEEE floating                           A
  - 64-bit IEEE floating                           B

The letters ABCD indicates decreasing "priority of use". In fact, talking 
of numeric values, there is usually little reason to prefer INTEGER values 
(with the annoyance of scaling) to REAL values, except perhaps sometimes
for space reasons (or for "conservative" reasons, see below).
E.g. if I want to store a spectrum, I store it in counts/s (REAL) with
associated errors (REAL) and not in counts (INTEGER), etc.

-------------

One argument which was put forward by some contributors, and which I
term here for brevity "the conservative argument" is : "we should store
data as close as they come out from the instrument" (this arose, as far
as I remember, both from CCD users and from XTE)

i) this is sometimes reasonable, but cannot it be done within the
   existing conventions ?

   I often had to deal with 8-bit and 16-bit data from X-ray detectors.
   8-bit values might be some ADC channel level, or counts in an histogram 
   (and in both cases usually are in the full unsigned range 0-255).
   16-bit values may be counts in an histogram, but in this case although
   nominally unsigned the probability of being above 32767 is really low.
   I hardly see the need of using the full range of a 32-bit (signed or
   unsigned) for anything (but on-board clocks, which deserve a special
   treatment anyhow).

   One should separate the way the data are STORED in a file, and the
   way they are ELABORATED in variables in a program. As a matter of
   fact I operate only on INTEGER*4 variables. 
   I note that FITS (both the "original" FITS and the Binary Tables)
   consider 8-bit values to be unsigned (unlike other integers, and
   unlike the convention for the non-standard BYTE data type in some 
   Fortrans; in fact I never use the BYTE data type, but always read
   the unsigned 8-bit value in an INTEGER*4 with some  CHAR/ICHAR
   trick). 
   Copying the two bytes of an INTEGER*2 into an INTEGER*4 preserving
   its value should be reasonably easy with some EQUIVALENCE tricks.
   
   I refer also to previous postings about BSCALE, BZERO etc.

ii) there are cases in which the conservative argument lead to the
    extrema would cause proliferation of strange data types.

   It often happens that satellite data have a "funny" number of bits
   (either because they are "naturally" produced this way in the 
   electronics, or because the need of the telemetry require to chop
   off redundant 0 bits) : so one can have a 10-bit energy, a 3-bit
   detector id, etc.

   In this case I would consider extremely annoying for the general
   observer to deal with such data types. However that is what would
   be implied if one follows strictly the "store data as they come
   out of the instrument" approach.
   For instance, the requirement which was put on the Ground Segment
   Contractor for SAX, was to reformat such data in a simple way, i.e.
   expand all 1- to 7-bit fields into an 8-bit 0-padded field, all
   9- to 15-bit fields into a 16-bit field, and all 17- to 31-bit
   fields into a 32-bit field (the "natural" sizes which can be
   handled as said in (i) above).
   This way data are kept "close to the form they come out of the
   instrument" (e.g. the partitioning in telemetry packets is
   preserved, and no destructive processing is performed), but they
   are reformatted in a friendly way.

   I note that FITS does not consider a true bit data type, since the
   rX binary table type requires the same padding to n*8 bit, but
   unfortunately (unexplainably ?) requires left-justification.

iii) a philosophical argument : when is something to be called FITS.

   As far as I remember the postings about XTE were mentioning "telemetry
   data buffers" downlinked from the spacecraft in a "huge number" (quoted
   as "dozens" to "thousands" sic!) "of data types", which had to be
   "stored as close as possible as they come down from the spacecraft",
   but at the same time be FITS. And the proposed solution was to store
   each "buffer" as a row in a FITS table.

   Now, while I could see good reasons to keep the "buffers" (more or
   less corresponding to the "telemetry packets" mentioned in ii above)
   as untouched as possible (but I hope this does not extend to keeping
   5-bit fields ... again see ii above), what I fail to see is why one
   should try to encapsulate this which is not (and by good reasons)
   a FITS structure into a "nominalistic" FITS wrapper.

   If one had to do some processing, and export processed data, I would
   agree about the case for using an existing FITS structure, or
   devising an easy one.

   But in this and similar cases, I find strange the following line of
   reasoning :

      - Files must be written in FITS
      - My data are not compatible with FITS
      - The FITS convention must be changed so that my data can
        be enclosed unchanged 
        and the result still be called FITS, q.e.d.

   This seems to me a strange argument, like I were saying :

   - English uses the latin alphabet
   - All documents must be written in English
   - My "data" are written in Hungarian.
   - I write my document putting in each page a heading saying "what
     follows is hungarian text" followed by one page of the original
     text. Of course I use the latin alphabet.
   - The document is written in English, q.e.d.
 
---------------------------------------------------------------------------- 
       A member of  G.ASS : Group for Astronomical Software Support          
---------------------------------------------------------------------------- 
Lucio Chiappetti - IFCTR/CNR     | Ma te' vugl' da' quost avis a ti' Orsign  
via Bassini 15 - I-20133 Milano  | Buttet rabios intant te se' pisnign       
Internet: LUCIO at IFCTR.MI.CNR.IT  |                                           
Decnet:   IFCTR::LUCIO           |             (Rabisch, II 46, 119-120)     
---------------------------------------------------------------------------- 

From fitsbits-request  Fri Mar 18 16:58:49 1994
Status: RO
X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil]
	["202" "" "18" "March" "1994" "16:56:12" "GMT" "Tom Lipovits" "ac448 at cleveland.freenet.edu" "<2mcmfc$g3d at usenet.INS.CWRU.Edu>" "7" "Is there a FITS image viewer for DOS?" "^From:" nil nil "3" "1994031816:56:12" "Is there a FITS image viewer for DOS?" (number " " mark "     Tom Lipovits      Mar 18    7/202   " thread-indent "\"Is there a FITS image viewer for DOS?\"\n") nil]
	nil)
Received: by fits.cv.nrao.edu (4.1/DDN-DLB/1.5)
	id AA20108; Fri, 18 Mar 94 16:58:49 EST
Return-Path: <news at saips.CV.NRAO.EDU>
Message-Id: <2mcmfc$g3d at usenet.INS.CWRU.Edu>
Organization: Case Western Reserve University, Cleveland, Ohio (USA)
Path: saips.cv.nrao.edu!hearst.acc.Virginia.EDU!concert!news.duke.edu!MathWorks.Com!europa.eng.gtefsd.com!howland.reston.ans.net!usenet.ins.cwru.edu!cleveland.Freenet.Edu!ac448
From: ac448 at cleveland.Freenet.Edu (Tom Lipovits)
Sender: fitsbits-request at fits.CV.NRAO.EDU
To: fitsbits at fits.CV.NRAO.EDU
Subject: Is there a FITS image viewer for DOS?
Date: 18 Mar 1994 16:56:12 GMT

I am recently  interested in the FITS images and am unable to find
any image viewers on local bulletin boards.  Any information would
be helpful.  Thank you
   
    _I_don't_have_an_employer_
   .Tom.

From fitsbits-request  Sat Mar 19 23:00:54 1994
Status: RO
X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil]
	["7008" "" "20" "March" "1994" "03:43:43" "GMT" "Martin Shepherd" "mcs at goblin.caltech.edu" "<MCS.94Mar19194343 at goblin.caltech.edu>" "145" "Re: Unsigned Integers Proposal" "^From:" nil nil "3" "1994032003:43:43" "Unsigned Integers Proposal" (number " " mark "     Martin Shepherd   Mar 20  145/7008  " thread-indent "\"Re: Unsigned Integers Proposal\"\n") "<9403181638.AA11251 at xebec.gsfc.nasa.gov>"]
	nil)
Received: by fits.cv.nrao.edu (4.1/DDN-DLB/1.5)
	id AA21952; Sat, 19 Mar 94 23:00:54 EST
Return-Path: <news at saips.CV.NRAO.EDU>
Message-Id: <MCS.94Mar19194343 at goblin.caltech.edu>
Organization: California Institute of Technology.
Path: saips.cv.nrao.edu!hearst.acc.Virginia.EDU!caen!usenet.cis.ufl.edu!usenet.ufl.edu!gatech!swrinde!elroy.jpl.nasa.gov!netline-fddi.jpl.nasa.gov!nntp-server.caltech.edu!nntp-server.caltech.edu!mcs
References: <9403181638.AA11251 at xebec.gsfc.nasa.gov>
From: mcs at goblin.caltech.edu (Martin Shepherd)
Sender: fitsbits-request at fits.CV.NRAO.EDU
To: fitsbits at fits.CV.NRAO.EDU
Subject: Re: Unsigned Integers Proposal
Date: 20 Mar 1994 03:43:43 GMT

In article <9403181638.AA11251 at xebec.gsfc.nasa.gov> arots at xebec.gsfc.nasa.gov (Arnold Rots) writes:
:I have watched the debate over the issue of introducing unsigned
:integers into the FITS Binary Tables closely and my conclusion (not
:surprising, since I started this round in the discussion) is: we
:should do it and we'd better do it quickly.

I have also been watching the debate, but have come to the opposite
conclusion. My personal viewpoint is that requiring FITS reader
software to become more complex just so that a few type-specific FITS
writers can be written more easily is totally unreasonable.

:The Binary Table standard contains a fairly large and fairly complete
:set of data types.  The criterion for defining this set cannot have
:been to arrive at a minimum set.  Considering the fact that it
:contains complex and double complex, the approach must have been to
:include all meaningful data types commonly in use among astronomers.
:I would submit that unsigned integers ought to be included in this
:collection, simply on the basis of generality.

:Frankly, I think, there is too much parochial thinking going on.

This goes both ways. Requiring that FITS be molded to your specialized
needs without regard to the problems that it would produce for the
rest of the community is surely a parochial viewpoint.

:  I
:have no intention to pick on anyone in particular, but I understand
:very well where Don Wells is coming from: in radio astronomy floats,
:doubles, complex, signed integers and scaled integers have
:traditionally dominated the zoo of data types.

This is unreasonable. It seems far more likely that the chosen types
were based on those then available in FORTRAN-77 rather than a
conspiracy among radio astronomers. While I personally prefer C, I can
see the point of not supporting types that at least one language can't
handle. Admittedly, from this viewpoint, COMPLEX types should not have
been included, but it is undeniably too late to change that now.

:[Text omitted]

:Again, if we *can* store original values in FITS tables, I feel it
:*should* be done.

In an ideal world perhaps, but in the real world it is pointless
because many architectures don't provide types with the same bit
representation as FITS decrees, and are thus unable to recover the
exact value that was written.

At a time in which object oriented programming is showing such
promise, isn't it becoming clear that the particular form in which
data are stored is secondary to the interface used to access them?
FITS allows such abstraction, and should be commended for this. If you
care about how your data is represented at the bit level then don't
use FITS - it wasn't designed for that.

:  Every extra line of code increases complexity and
:chances for errors; this is of specific concern to those among us who
:work on software that will produce basic archives: the number of
:operations should be kept to an absolute minimum, on the one hand,
:while the archived product should provide as good data access as
:possible.  Unsigned integers would help tremendously in that regard.

Again the same argument can be presented from the opposite viewpoint.
Adding new types increases the size and complexity of general FITS
readers. You are requiring that FITS readers be slowed down so
that your simpler, type-specific FITS writer can run faster. To make
this point clearer, consider that in a general FITS reader every new
type must be accompanied by at least three specialized sets of
functions:

1. Machine-specific routines to convert between each FITS type and
   equivalent host-architecture data-types. This makes it hard to
   port to new architectures - more so if there are a lot of types.

2. Conversion functions between all other compatible types supported
   by the FITS library (eg. int to unsigned int, int to float...).
   For N types there are O(NxN) such conversions required. This has a
   significant impact on speed and complexity.

3. Functions to test for and set type-specific blank/null values.

Obviously the more functions there are and the greater the number of
possible routes through a FITS reader, the more difficult it becomes to
write a general FITS library, the harder it is to test the result and
the slower it becomes to read FITS. This is the impact on the rest of
the community that I was referring to above, in turning your accusation
of parochial thinking back on you.

:[Text omitted]

:On to the XTE case.  Don Wells and Tom McGlynn have provided us with
:some quick calculations that all of us (I hope) are capable of
:carrying out but that, in their simplicity, completely miss the point.
:The problem is not in the actual scaling itself, though I might point
:out that we will decline Don's offer of his PC/XT clone, since we hope
:to be able to process a day's worth of telemetry in considerably less
:than 24 hours.  Figuring out when, where, and how to scale, involves at
:least the same amount of processing.

This makes it sound as though XTE was designed without regard for how
its output would be pipe-lined? Should the rest of the community have
to shoulder the burden of such a design flaw? Can the satellite
software not be changed to better match your archival requirements?

:More important, though, is the added complexity.  Of course, it can be
:determined what needs to be scaled by what.  And, of course, it can be
:table driven.  But we are talking about a significant increase in code
:complexity.

And should the rest of the community have to make their software more
complex so that your's need not be?

:[Text omitted]
:that capability; strict encapsulation and design simplicity have made
:this piece of software highly reliable - I would like to keep it that
:way.

I feel the same way about my general FITS library, and since it is
much easier to write a specific form of FITS than to read the general
case, shouldn't the burden be taken by the former? After all, during
data processing, FITS files often have to be read multiple times, so
speed and correctness during reading is very important to the
end-user.

:[More text omitted]
:I sincerely hope that the community will move forward with this
:expeditiously.  As Bill Pence remarked, this problem is not going
:away; therefore, the sooner we deal with it, the better.  And to
:require everybody to deal with the lack of unsigned integers in the
:future, because "we have always managed to do without" in the past,
:is, in my view, a dangerous position.

Given that computers are getting faster all the time, and that storage
facilities are getting more capacious, the argument for unsigned types
may soon be outdated. I can't see how it could get more pressing with
time. Unfortunately the same can't be said for 64 bit integers, but at
least they only represent one extra type to add.

:  - Arnold Rots

No offense intended. I just hope that you can find a simpler way to
fix XTE without requiring radical changes to FITS.

Martin Shepherd  (mcs at phobos.caltech.edu)

From fitsbits-request  Mon Mar 21 04:31:49 1994
Status: RO
X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil]
	["460" "Mon" "21" "March" "1994" "09:12:35" "GMT" "John Boyer" "boyer at rd.eng.bbc.co.uk" "<Cn0C8z.Knr at bbc.co.uk>" "16" "Re: Is there a FITS image viewer for DOS?" "^From:" nil nil "3" "1994032109:12:35" "Is there a FITS image viewer for DOS?" (number " " mark "     John Boyer        Mar 21   16/460   " thread-indent "\"Re: Is there a FITS image viewer for DOS?\"\n") "<2mcmfc$g3d at usenet.INS.CWRU.Edu>"]
	nil)
Received: by fits.cv.nrao.edu (4.1/DDN-DLB/1.5)
	id AA24594; Mon, 21 Mar 94 04:31:49 EST
Return-Path: <news at saips.CV.NRAO.EDU>
Message-Id: <Cn0C8z.Knr at bbc.co.uk>
Organization: British Broadcasting Corporation, UK
Path: saips.cv.nrao.edu!hearst.acc.Virginia.EDU!caen!sdd.hp.com!cs.utexas.edu!howland.reston.ans.net!pipex!bbc!ant!boyer
References: <2mcmfc$g3d at usenet.INS.CWRU.Edu>
From: boyer at rd.eng.bbc.co.uk (John Boyer)
Sender: fitsbits-request at fits.CV.NRAO.EDU
To: fitsbits at fits.CV.NRAO.EDU
Subject: Re: Is there a FITS image viewer for DOS?
Date: Mon, 21 Mar 1994 09:12:35 GMT

Tom Lipovits (ac448 at cleveland.Freenet.Edu) wrote:

: I am recently  interested in the FITS images and am unable to find
: any image viewers on local bulletin boards.  Any information would
: be helpful.  Thank you
:    
:     _I_don't_have_an_employer_
:    .Tom.
Yes. it's called imdisp79. i think I got it from ames.arc.nasa.gov. You
could also try explorer.arc.nasa.gov   It will not do 32 bit floating
point numbers. 

John B
John.boyer at rd.eng.bbc.co.uk

From fitsbits-request  Mon Mar 21 09:29:00 1994
Status: RO
X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil]
	["5486" "Mon" "21" "March" "1994" "09:28:52" "EST" "Arnold Rots" "arots at xebec.gsfc.nasa.gov" "<9403211428.AA12356 at xebec.gsfc.nasa.gov>" "112" "Re: Unsigned Integers Proposal" "^From:" nil nil "3" "1994032114:28:52" "Unsigned Integers Proposal" (number " " mark "     Arnold Rots       Mar 21  112/5486  " thread-indent "\"Re: Unsigned Integers Proposal\"\n") nil]
	nil)
Received: by fits.cv.nrao.edu (4.1/DDN-DLB/1.5)
	id AA25770; Mon, 21 Mar 94 09:29:00 EST
Return-Path: <arots at xebec.gsfc.nasa.gov>
Message-Id: <9403211428.AA12356 at xebec.gsfc.nasa.gov>
From: arots at xebec.gsfc.nasa.gov (Arnold Rots)
Sender: fitsbits-request at fits.CV.NRAO.EDU
To: mcs at goblin.caltech.edu
Cc: fitsbits at fits.CV.NRAO.EDU
Subject: Re: Unsigned Integers Proposal
Date: Mon, 21 Mar 94 09:28:52 EST

Martin Shepherd posted a detailed response to my last message.  I
won't replay to every point, but there are some issues that require
clarification.

My basic point was (and is) that the need for unsigned integer types
is fairly general and will be increasing in the future.  It's going to
be easier to introduce them now than n years from today.  The XTE
project is not the reason for our wanting to have unsigned integers
added to the list: I think FITS needs them and high-lighting their
usefulness in the context of the XTE software was intended to
illustrate how and why these data types are important and will
continue to be so in other projects.  The other side of the coins,
though, is that I have no great desire to jump through all kinds of
hoops now, if unsigned integers are going to be included in FITS
anyway.

> :  I
> :have no intention to pick on anyone in particular, but I understand
> :very well where Don Wells is coming from: in radio astronomy floats,
> :doubles, complex, signed integers and scaled integers have
> :traditionally dominated the zoo of data types.

> This is unreasonable. It seems far more likely that the chosen types
> were based on those then available in FORTRAN-77 rather than a
> conspiracy among radio astronomers. While I personally prefer C, I can
> see the point of not supporting types that at least one language can't
> handle. Admittedly, from this viewpoint, COMPLEX types should not have
> been included, but it is undeniably too late to change that now.

Sorry if this offended you.  However, I was a lot closer to the birth
of FITS than you were.  And I wasn't saying it was a conspiracy; I
said it was designed (with an honest effort to generalize) from a
specific set of needs.  That set did not include unsigned integers,
but it did include complex floats (case in point).

> :Again, if we *can* store original values in FITS tables, I feel it
> :*should* be done.

> In an ideal world perhaps, but in the real world it is pointless
> because many architectures don't provide types with the same bit
> representation as FITS decrees, and are thus unable to recover the
> exact value that was written.

Sorry, I wrote *values*, not *bit patterns*.

> :  Every extra line of code increases complexity and
> :chances for errors; this is of specific concern to those among us who
> :work on software that will produce basic archives: the number of
> :operations should be kept to an absolute minimum, on the one hand,
> :while the archived product should provide as good data access as
> :possible.  Unsigned integers would help tremendously in that regard.

> Again the same argument can be presented from the opposite viewpoint.
> Adding new types increases the size and complexity of general FITS
> readers. You are requiring that FITS readers be slowed down so
> that your simpler, type-specific FITS writer can run faster. To make
> this point clearer, consider that in a general FITS reader every new
> type must be accompanied by at least three specialized sets of
> functions:

> 1. Machine-specific routines to convert between each FITS type and
>    equivalent host-architecture data-types. This makes it hard to
>    port to new architectures - more so if there are a lot of types.

> 2. Conversion functions between all other compatible types supported
>    by the FITS library (eg. int to unsigned int, int to float...).
>    For N types there are O(NxN) such conversions required. This has a
>    significant impact on speed and complexity.

> 3. Functions to test for and set type-specific blank/null values.

> Obviously the more functions there are and the greater the number of
> possible routes through a FITS reader, the more difficult it becomes to
> write a general FITS library, the harder it is to test the result and
> the slower it becomes to read FITS. This is the impact on the rest of
> the community that I was referring to above, in turning your accusation
> of parochial thinking back on you.

Bill Pence addressed these issues in an earlier post and concluded
they weren't as significant as you make them appear.

> And should the rest of the community have to make their software more
> complex so that your's need not be?

As I said in the introduction, I was not trying to argue that the
whole community should change its standards because of XTE, only that
many projects (XTE and the guys at FNAL are a case in point) would
benefit tremendously from this proposal.

> :[Text omitted]
> :that capability; strict encapsulation and design simplicity have made
> :this piece of software highly reliable - I would like to keep it that
> :way.

> I feel the same way about my general FITS library, and since it is
> much easier to write a specific form of FITS than to read the general
> case, shouldn't the burden be taken by the former? After all, during
> data processing, FITS files often have to be read multiple times, so
> speed and correctness during reading is very important to the
> end-user.

This raises an interesting question on one's fundamental design
approach: should basic utility libraries (like FITS readers) be kept
simple so that they are reliable, but requiring the application
software that uses them to do the work and be more complex; or should
those libraries be designed in such a way that the user-software can
be simpler?  Whatever case can be made against the introduction of
unsigned integers, I don't think it could be supported by this
argument.

  - Arnold Rots

From fitsbits-request  Mon Mar 21 12:22:09 1994
Status: RO
X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil]
	["1234" "" "21" "March" "1994" "16:48:57" "GMT" "Tom Nicinski" "nicinski at fncasefnal.fnal.gov" "<2mkj5p$nno at fnnews.fnal.gov>" "25" "Re: Unsigned Integers Proposal" "^From:" nil nil "3" "1994032116:48:57" "Unsigned Integers Proposal" (number " " mark "     Tom Nicinski      Mar 21   25/1234  " thread-indent "\"Re: Unsigned Integers Proposal\"\n") "<9403211428.AA12356 at xebec.gsfc.nasa.gov>"]
	nil)
Received: by fits.cv.nrao.edu (4.1/DDN-DLB/1.5)
	id AA26131; Mon, 21 Mar 94 12:22:09 EST
Return-Path: <news at saips.CV.NRAO.EDU>
Message-Id: <2mkj5p$nno at fnnews.fnal.gov>
Organization: FERMILAB, Batavia, IL
Path: saips.cv.nrao.edu!hearst.acc.Virginia.EDU!concert!news.duke.edu!MathWorks.Com!europa.eng.gtefsd.com!howland.reston.ans.net!news.moneng.mei.com!uwm.edu!fnnews.fnal.gov!fncase!nicinski
References: <9403211428.AA12356 at xebec.gsfc.nasa.gov>
Reply-To: nicinski at fnal.fnal.gov
From: nicinski at fncasefnal.fnal.gov (Tom Nicinski)
Sender: fitsbits-request at fits.CV.NRAO.EDU
To: fitsbits at fits.CV.NRAO.EDU
Subject: Re: Unsigned Integers Proposal
Date: 21 Mar 1994 16:48:57 GMT

arots at xebec.gsfc.nasa.gov (Arnold Rots) writes:

[deleted ...]

|> My basic point was (and is) that the need for unsigned integer types
|> is fairly general and will be increasing in the future.  It's going to
|> be easier to introduce them now than n years from today.

I'd also like to see unsigned integers introduced soon as part of the
BINTABLE extension, before it changes from a draft to a standard.  Letting
this issue sit and then providing another "standard" extension that is the
BINTABLE extension plus unsigned integers (and signed 8-bit integers and
possibly 64-bit integers (signed and unsigned)) would be a mistake.  Talk
about making FITS more complex:  you'd have to worry about maintaining two
extensions in synch with each other.

Tom Nicinski

+--- . .' ----+--------------------------------+----------------------------+
|   /| | |  | | Tom Nicinski / MS 234          | nicinski at fnal.fnal.gov     |
|    +-+-+\ | | Fermi National Accelerator Lab |                         _  |
| .  | | | \| | P.O. Box 500                   |                        [o] |
| `--' | |  | | Batavia, IL 60510-0500         |                        /|\ |
+---- -' -----+--------------------------------+----------------------------+

From fitsbits-request  Mon Mar 21 12:59:11 1994
Status: RO
X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil]
	["2905" "Mon" "21" "March" "1994" "18:56:25" "+0100" "Thierry Forveille" "forveill at gag.observ-gr.fr" "<9403211758.AA26156 at fits.cv.nrao.edu>" "53" "RE: Unsigned integer proposal" "^From:" nil nil "3" "1994032117:56:25" "Unsigned integer proposal" (number " " mark "     Thierry Forveille Mar 21   53/2905  " thread-indent "\"RE: Unsigned integer proposal\"\n") nil]
	nil)
Received: by fits.cv.nrao.edu (4.1/DDN-DLB/1.5)
	id AA26162; Mon, 21 Mar 94 12:59:11 EST
Return-Path: <forveill at gag.observ-gr.fr>
Message-Id: <9403211758.AA26156 at fits.cv.nrao.edu>
Posted-Date: Mon, 21 Mar 1994 18:56:25 +0100
From: forveill at gag.observ-gr.fr (Thierry Forveille)
Sender: fitsbits-request at fits.CV.NRAO.EDU
To: fitsbits at fits.CV.NRAO.EDU
Subject: RE: Unsigned integer proposal
Date: Mon, 21 Mar 1994 18:56:25 +0100

------- Start of forwarded message -------
Posted-Date: Mon, 21 Mar 1994 18:32:42 +0100
In-Reply-To: <9403211428.AA12356 at xebec.gsfc.nasa.gov>
References: <9403211428.AA12356 at xebec.gsfc.nasa.gov>
From: forveill at gag.observ-gr.fr (Thierry Forveille)
To: arots at xebec.gsfc.nasa.gov (Arnold Rots)
Subject: Re: Unsigned Integers Proposal
Date: Mon, 21 Mar 1994 18:32:42 +0100

Arnold Rots writes:
 > 
 > My basic point was (and is) that the need for unsigned integer types
 > is fairly general and will be increasing in the future.  It's going to
 > be easier to introduce them now than n years from today.  The XTE
 > project is not the reason for our wanting to have unsigned integers
 > added to the list: I think FITS needs them and high-lighting their
 > usefulness in the context of the XTE software was intended to
 > illustrate how and why these data types are important and will
 > continue to be so in other projects.  The other side of the coins,
 > though, is that I have no great desire to jump through all kinds of
 > hoops now, if unsigned integers are going to be included in FITS
 > anyway.
 > 
I happen to hold just the opposite opinion: the (perceived) need for unsigned
integers is highly dated to a few years around 1993 and will fade away as it
has arrived. The difference between signed and unsigned integers only matters
when you don't fit on 15 bits, so nobody ever cared about FITS integers 
being signed before 16 bits ADC became common-place, and nobody will ever
care again once 18 or 20 bits ADC will be the norm (though I could imagine some
support gathered for 24 bit integers at that time...). Given typical evolution
times in the electronic industry, my prediction is that this is going to
happen before long, though the convenience of 16 bits as the half word of most
computers may delay that evolution by a year or two. One can already see that
need appearing in CCD manuals that state that the device is linear to better
than 1% over the full range of the ADC: that just means the ADC cannot use 
the full range of the CCD and still sample the noise correctly. What you are 
buying with unsigned integers is just one bit (which you could also get more 
cheaply through a BZERO) and this isn't going to last you very long. A new
data type in FITS should bring more than that (I for one would probably buy
64 bit integers).

	Thierry Forveille
	Observatoire de Grenoble

PS: Since this is apparently becoming an issue in the discussion, I should
probably state that I am a radioastronomer by training (is that a liability?), 
but now working on DENIS, a near IR survey of the sky which will produce about 
as many terabytes as the Sloan project (bigger pixels but more steradians and 
filters). The cost of flipping the high order bit was never even discussed and
is dwarfed  y other things like computing a decent median flat-field.
------- End of forwarded message -------

From fitsbits-request  Mon Mar 21 15:24:03 1994
Status: RO
X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil]
	["3307" "" "21" "March" "1994" "20:00:19" "GMT" "Tim Pearson" "tjp at bottom.caltech.edu" "<2mkucj$j3u at gap.cco.caltech.edu>" "59" "Re: Unsigned integer proposal" "^From:" nil nil "3" "1994032120:00:19" "Unsigned integer proposal" (number " " mark "     Tim Pearson       Mar 21   59/3307  " thread-indent "\"Re: Unsigned integer proposal\"\n") "<9403211758.AA26156 at fits.cv.nrao.edu>"]
	nil)
Received: by fits.cv.nrao.edu (4.1/DDN-DLB/1.5)
	id AA00308; Mon, 21 Mar 94 15:24:03 EST
Return-Path: <news at saips.CV.NRAO.EDU>
Message-Id: <2mkucj$j3u at gap.cco.caltech.edu>
Organization: California Institute of Technology, Pasadena
Path: saips.cv.nrao.edu!hearst.acc.Virginia.EDU!concert!news.duke.edu!MathWorks.Com!europa.eng.gtefsd.com!howland.reston.ans.net!cs.utexas.edu!swrinde!elroy.jpl.nasa.gov!netline-fddi.jpl.nasa.gov!nntp-server.caltech.edu!bottom!tjp
References: <9403211758.AA26156 at fits.cv.nrao.edu>
From: tjp at bottom.caltech.edu (Tim Pearson)
Sender: fitsbits-request at fits.CV.NRAO.EDU
To: fitsbits at fits.CV.NRAO.EDU
Subject: Re: Unsigned integer proposal
Date: 21 Mar 1994 20:00:19 GMT

I too am opposed to adding complexity to FITS readers by adding new
unsigned integer data types to the FITS standard.

The purpose of writing in FITS format is to simplify the transport of
data: that's what the T stands for. Files are always read many more
times than they are written (if not, you might question why you are
writing the file in the first place), and so when defining a data format
it is much more important to consider the needs of the reader than those
of the writer. If we change FITS now, many hours - probably many
hundreds of hours - will be devoted to modifying FITS-reading code to
cope. I question whether this is necessary when unsigned integers of up
to 32 bits can already be represented exactly in FITS files with no
change of standards.

"Unsigned" means that all the bits are used for magnitude, and none is
reserved for sign, and usually the range of values allowed is 0 to 
(2^n)-1, where n is the number of bits. In a FITS file, the range of
values represented may be different (owing to BSCALE and BZERO), and
really has to be determined by examining the data. This applies to both
the present FITS integer types and any proposed unsigned type. [As an
example: suppose a FITS file complying with existing standards specifies
BITPIX=16 and BSCALE=5000: the reader must not blindly multiply the
values by 5000 using 16-bit integer arithmetic!]

Unsigned integers up to 15 bits long can be stored in FITS 16-bit
integers with no problem; similarly unsigned integers from 16 to 31 bits
can be stored in FITS 32-bit integers. Problems only arise when trying 
to store 16-bit unsigned integers in FITS 16-bit integers or 32-bit
unsigned integers in FITS 32-bit integers. However, this can be done
with no loss of accuracy or range by using BZERO, as explained in
earlier contributions to this discussion. Problems may arise when the
reading program tries to apply the BZERO correction if the arithmetic is
not done correctly (e.g., assuming that the corrected result can be
stored in 16-bit signed integer), but that is a problem that FITS
readers already have to address.

"Unsigned" is sometimes (e.g., in the C language) used to mean that
arithmetic on these quantities is done modulo 2^n. I don't think it is
appropriate to impose this restriction on a FITS reader: the writer of the
file does not known what arithmetic is going to be done on the values
contained in the file, now what purpose they will be put to; his
responsibility is solely to ensure that the correct values (not bit
patterns) are read.

Finally, I question whether a particular hardware design (the use of a
16-bit A/D converter, for example) should drive software design,
particularly for data structures that are to be transported to a wide
community. Does the particular quantity being sampled with the A/D
converter really require exactly 1 part in 2^16 precision? Or would 1
part in 2^15 really suffice? Or perhaps you would prefer greater
precision, in which case you should maybe get an 18-bit A/D converter.
But please don't suggest we add 18-bit integer types to FITS.

Tim Pearson
-- 

Tim Pearson, Astronomy Dept 105-24, Caltech, Pasadena, California 91125, USA
Internet:                  TJP  at  Deimos.Caltech.Edu, Pearson_T  at  caltech.edu
NSI-Decnet (SPAN):         Deimos::TJP or 15237::TJP

From fitsbits-request  Mon Mar 21 19:01:30 1994
Status: RO
X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil]
	["2134" "" "21" "March" "1994" "23:38:28" "GMT" "Stephen A. Voels" "VOELS at mystry" "<2mlb5k$87j at paperboy.gsfc.nasa.gov>" "43" "RE: Unsigned integer proposal" "^From:" nil nil "3" "1994032123:38:28" "Unsigned integer proposal" (number " " mark "     Stephen A. Voels  Mar 21   43/2134  " thread-indent "\"RE: Unsigned integer proposal\"\n") nil]
	nil)
Received: by fits.cv.nrao.edu (4.1/DDN-DLB/1.5)
	id AA00584; Mon, 21 Mar 94 19:01:30 EST
Return-Path: <news at saips.CV.NRAO.EDU>
Message-Id: <2mlb5k$87j at paperboy.gsfc.nasa.gov>
Organization: NASA/GSFC/ADF
Path: saips.cv.nrao.edu!hearst.acc.Virginia.EDU!caen!math.ohio-state.edu!howland.reston.ans.net!cs.utexas.edu!swrinde!elroy.jpl.nasa.gov!ames!newsfeed.gsfc.nasa.gov!usenet
From: VOELS at MYSTRY (Stephen A. Voels)
Sender: fitsbits-request at fits.CV.NRAO.EDU
To: fitsbits at fits.CV.NRAO.EDU
Subject: RE: Unsigned integer proposal
Date: 21 Mar 1994 23:38:28 GMT

Thierry Forveille writes:
>
>Arnold Rots writes:
> > 
> > My basic point was (and is) that the need for unsigned integer types
> > is fairly general and will be increasing in the future.  It's going to
> > be easier to introduce them now than n years from today.  The XTE
> > project is not the reason for our wanting to have unsigned integers
> > added to the list: I think FITS needs them and high-lighting their
> > usefulness in the context of the XTE software was intended to
> > illustrate how and why these data types are important and will
> > continue to be so in other projects.  The other side of the coins,
> > though, is that I have no great desire to jump through all kinds of
> > hoops now, if unsigned integers are going to be included in FITS
> > anyway.
> > 
>I happen to hold just the opposite opinion: the (perceived) need for unsigned
>integers is highly dated to a few years around 1993 and will fade away as it
>has arrived.

NO.  This question has been around since the FIRST FITS paper -- I
personally have raised it every time a modification or extension has been
proposed or made to FITS.  There have been and are currently several ground
based CCD projects which could use this.  And I am willing to bet that many
of these projects will be around for several years -- not every group
updates or wants to update their equipement when something better comes
along and older devices are often "willed" to insitutes with less
resources.

It is long past time to put these data types in and now is as good a time
as any.  I know too many projects which decided to go with SIMPLE=F just
because of this.

It does make a difference when you want simple programs at the data taking
location that store the data AS GENERATED.  That is THE best way to make
sure data archived directly from the detectors is correct.

If the FITS groups so not want to recognize this, I personally think Arnold
should just go ahead and define his own extension eventhough it may be
close to BINTABLE and do what is best for his project.  With that much data
people will follow, plus other groups are sure to use it.

Stephen A. Voels

From fitsbits-request  Mon Mar 21 19:01:28 1994
Status: RO
X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil]
	["5221" "" "21" "March" "1994" "23:37:33" "GMT" "Stephen A. Voels" "VOELS at mystry" "<2mlb3t$87j at paperboy.gsfc.nasa.gov>" "98" "RE: Unsigned integer proposal" "^From:" nil nil "3" "1994032123:37:33" "Unsigned integer proposal" (number " " mark "     Stephen A. Voels  Mar 21   98/5221  " thread-indent "\"RE: Unsigned integer proposal\"\n") "<9403211758.AA26156 at fits.cv.nrao.edu>"]
	nil)
Received: by fits.cv.nrao.edu (4.1/DDN-DLB/1.5)
	id AA00577; Mon, 21 Mar 94 19:01:28 EST
Return-Path: <news at saips.CV.NRAO.EDU>
Message-Id: <2mlb3t$87j at paperboy.gsfc.nasa.gov>
Organization: NASA/GSFC/ADF
Path: saips.cv.nrao.edu!hearst.acc.Virginia.EDU!caen!math.ohio-state.edu!howland.reston.ans.net!cs.utexas.edu!swrinde!elroy.jpl.nasa.gov!ames!newsfeed.gsfc.nasa.gov!usenet
References: <9403211758.AA26156 at fits.cv.nrao.edu>
From: VOELS at MYSTRY (Stephen A. Voels)
Sender: fitsbits-request at fits.CV.NRAO.EDU
To: fitsbits at fits.CV.NRAO.EDU
Subject: RE: Unsigned integer proposal
Date: 21 Mar 1994 23:37:33 GMT

In <9403211758.AA26156 at fits.cv.nrao.edu> forveill at gag.observ-gr.fr writes:

> ------- Start of forwarded message -------
> Posted-Date: Mon, 21 Mar 1994 18:32:42 +0100
> In-Reply-To: <9403211428.AA12356 at xebec.gsfc.nasa.gov>
> References: <9403211428.AA12356 at xebec.gsfc.nasa.gov>
> From: forveill at gag.observ-gr.fr (Thierry Forveille)
> To: arots at xebec.gsfc.nasa.gov (Arnold Rots)
> Subject: Re: Unsigned Integers Proposal
> Date: Mon, 21 Mar 1994 18:32:42 +0100
> 
> Arnold Rots writes:
>  > 
>  > My basic point was (and is) that the need for unsigned integer types
>  > is fairly general and will be increasing in the future.  It's going to
>  > be easier to introduce them now than n years from today.  The XTE
>  > project is not the reason for our wanting to have unsigned integers
>  > added to the list: I think FITS needs them and high-lighting their
>  > usefulness in the context of the XTE software was intended to
>  > illustrate how and why these data types are important and will
>  > continue to be so in other projects.  The other side of the coins,
>  > though, is that I have no great desire to jump through all kinds of
>  > hoops now, if unsigned integers are going to be included in FITS
>  > anyway.
>  > 
> I happen to hold just the opposite opinion: the (perceived) need for unsigned
> integers is highly dated to a few years around 1993 and will fade away as it
> has arrived. The difference between signed and unsigned integers only matters
> when you don't fit on 15 bits, so nobody ever cared about FITS integers 
> being signed before 16 bits ADC became common-place, and nobody will ever
> care again once 18 or 20 bits ADC will be the norm (though I could imagine some
> support gathered for 24 bit integers at that time...). Given typical evolution
> times in the electronic industry, my prediction is that this is going to
> happen before long, though the convenience of 16 bits as the half word of most
> computers may delay that evolution by a year or two. One can already see that
> need appearing in CCD manuals that state that the device is linear to better
> than 1% over the full range of the ADC: that just means the ADC cannot use 
> the full range of the CCD and still sample the noise correctly. What you are 
> buying with unsigned integers is just one bit (which you could also get more 
> cheaply through a BZERO) and this isn't going to last you very long. A new
> data type in FITS should bring more than that (I for one would probably buy
> 64 bit integers).
> 
> 	Thierry Forveille
> 	Observatoire de Grenoble
> 
> PS: Since this is apparently becoming an issue in the discussion, I should
> probably state that I am a radioastronomer by training (is that a liability?), 
> but now working on DENIS, a near IR survey of the sky which will produce about 
> as many terabytes as the Sloan project (bigger pixels but more steradians and 
> filters). The cost of flipping the high order bit was never even discussed and
> is dwarfed  y other things like computing a decent median flat-field.
> ------- End of forwarded message -------
> 
> 
Thierry Forveille writes:
>
>Arnold Rots writes:
> > 
> > My basic point was (and is) that the need for unsigned integer types
> > is fairly general and will be increasing in the future.  It's going to
> > be easier to introduce them now than n years from today.  The XTE
> > project is not the reason for our wanting to have unsigned integers
> > added to the list: I think FITS needs them and high-lighting their
> > usefulness in the context of the XTE software was intended to
> > illustrate how and why these data types are important and will
> > continue to be so in other projects.  The other side of the coins,
> > though, is that I have no great desire to jump through all kinds of
> > hoops now, if unsigned integers are going to be included in FITS
> > anyway.
> > 
>I happen to hold just the opposite opinion: the (perceived) need for unsigned
>integers is highly dated to a few years around 1993 and will fade away as it
>has arrived.

NO.  This question has been around since the FIRST FITS paper -- I
personally have raised it every time a modification or extension has been
proposed or made to FITS.  There have been and are currently several ground
based CCD projects which could use this.  And I am willing to bet that many
of these projects will be around for several years -- not every group
updates or wants to update their equipement when something better comes
along and older devices are often "willed" to insitutes with less
resources.

It is long past time to put these data types in and now is as good a time
as any.  I know too many projects which decided to go with SIMPLE=F just
because of this.

It does make a difference when you want simple programs at the data taking
location that store the data AS GENERATED.  That is THE best way to make
sure data archived directly from the detectors is correct.

If the FITS groups so not want to recognize this, I personally think Arnold
should just go ahead and define his own extension eventhough it may be
close to BINTABLE and do what is best for his project.  With that much data
people will follow, plus other groups are sure to use it.

Stephen A. Voels

From fitsbits-request  Tue Mar 22 03:50:47 1994
Status: RO
X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil]
	["176" "" "22" "March" "1994" "02:17:55" "-0600" "T. C. Zhao" "svec5 at menudo.uh.edu" "<2mm9jj$kkq at menudo.uh.edu>" "8" "3D fits files ?" "^From:" nil nil "3" "1994032208:17:55" "3D fits files ?" (number " " mark "     T. C. Zhao        Mar 22    8/176   " thread-indent "\"3D fits files ?\"\n") nil]
	nil)
Received: by fits.cv.nrao.edu (4.1/DDN-DLB/1.5)
	id AA01081; Tue, 22 Mar 94 03:50:47 EST
Return-Path: <news at saips.CV.NRAO.EDU>
Message-Id: <2mm9jj$kkq at menudo.uh.edu>
Organization: University of Houston
Path: saips.cv.nrao.edu!hearst.acc.Virginia.EDU!darwin.sura.net!spool.mu.edu!agate!howland.reston.ans.net!cs.utexas.edu!swrinde!menudo.uh.edu!not-for-mail
From: svec5 at menudo.uh.edu (T.C. Zhao)
Sender: fitsbits-request at fits.CV.NRAO.EDU
To: fitsbits at fits.CV.NRAO.EDU
Subject: 3D fits files ?
Date: 22 Mar 1994 02:17:55 -0600

Hi,
Can someone tell me an ftp site where I can find some fits
files with NAXIS=3 ? floating point/double data would be
nice.

Thanks in advance.
-- 
T.C. - Starving Physicist

From fitsbits-request  Tue Mar 22 03:49:13 1994
Status: RO
X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil]
	["3642" "" "22" "March" "1994" "08:34:34" "GMT" "Martin Shepherd" "mcs at goblin.caltech.edu" "<MCS.94Mar22003434 at goblin.caltech.edu>" "81" "Re: Unsigned Integers Proposal" "^From:" nil nil "3" "1994032208:34:34" "Unsigned Integers Proposal" (number " " mark "     Martin Shepherd   Mar 22   81/3642  " thread-indent "\"Re: Unsigned Integers Proposal\"\n") "<9403211428.AA12356 at xebec.gsfc.nasa.gov>"]
	nil)
Received: by fits.cv.nrao.edu (4.1/DDN-DLB/1.5)
	id AA01072; Tue, 22 Mar 94 03:49:13 EST
Return-Path: <news at saips.CV.NRAO.EDU>
Message-Id: <MCS.94Mar22003434 at goblin.caltech.edu>
Organization: California Institute of Technology.
Path: saips.cv.nrao.edu!hearst.acc.Virginia.EDU!caen!spool.mu.edu!howland.reston.ans.net!cs.utexas.edu!swrinde!elroy.jpl.nasa.gov!netline-fddi.jpl.nasa.gov!nntp-server.caltech.edu!nntp-server.caltech.edu!mcs
References: <9403211428.AA12356 at xebec.gsfc.nasa.gov>
From: mcs at goblin.caltech.edu (Martin Shepherd)
Sender: fitsbits-request at fits.CV.NRAO.EDU
To: fitsbits at fits.CV.NRAO.EDU
Subject: Re: Unsigned Integers Proposal
Date: 22 Mar 1994 08:34:34 GMT

In article <9403211428.AA12356 at xebec.gsfc.nasa.gov> arots at xebec.gsfc.nasa.gov (Arnold Rots) writes:
:Martin Shepherd posted a detailed response to my last message.  I
:won't replay to every point, but there are some issues that require
:clarification.

:My basic point was (and is) that the need for unsigned integer types
:is fairly general and will be increasing in the future.  It's going to
:be easier to introduce them now than n years from today.
:...

I understand your viewpoint, but most of the responses that I have
seen contained objections to the proposal, so it was a surprise to see
anybody coming to the conclusion from the discussion, that unsigned
types should be implemented forthwith.

I do agree that, *if* unsigned types have to be added to FITS then
agreement should be reached on how to do it soon, and *if* they are to
be implemented then for the sake of conciseness and simplicity they
should be added to BINTABLE rather than by designing another derived
extension type, as one poster suggested. So far the consensus doesn't
seem to support such a change.

:>[Discussion of how adding unsigned types to FITS would complicate
   general FITS libraries more than not adding them would hurt the
   few specialized programs that find the conversion from unsigned
   to signed+offset to be a bottle-neck].

:Bill Pence addressed these issues in an earlier post and concluded
:they weren't as significant as you make them appear.

Regardless, they are more significant than simply abstracting unsigned
types via a functional interface to existing signed FITS types.

:>[Discussion of where the burden of handling unsigned types should lie]

:This raises an interesting question on one's fundamental design
:approach: should basic utility libraries (like FITS readers) be kept
:simple so that they are reliable, but requiring the application
:software that uses them to do the work and be more complex; or should
:those libraries be designed in such a way that the user-software can
:be simpler?  Whatever case can be made against the introduction of
:unsigned integers, I don't think it could be supported by this
:argument.

I agree that general FITS readers should hide as much of the
implementation as possible, but there are limits. FITS is already
fairly complicated to write a general library for and difficult to
implement correctly, efficiently, and portably with the current zoo of
ad hoc types and rules.

In places where support for unsigned types is desirable, FITS
libraries can easily use encapsulation to hide the fact that
unsigned types are implemented as a signed type and an offset, without
the user ever having to deal with the gory details. This achieves the
same effect without entailing as much extra code as handling unsigned
types at the raw FITS byte level.

My understanding is that proponents of adding unsigned types,
object to abstracting them through existing FITS types because:

1. The required offset to convert unsigned to signed, changes the
   value written.

 Why is this a problem. A simple scalar addition recovers the exact
 values that were written. By the time the user of any reasonable FITS
 reader gets the data, the values will have been restored to their
 original form.

2. It takes time to offset the unsigned type when recording it as
   a signed type.

 I find it hard to believe that this could have any significance
 compared to the overhead of file I/O.

Additionally, in the special case of XTE the argument seems to be that
it is difficult to determine what is being sent by the satellite? This
is hardly a FITS problem.

Martin Shepherd  (mcs at phobos.caltech.edu)

From fitsbits-request  Tue Mar 22 12:46:30 1994
Status: RO
X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil]
	["2312" "Tue" "22" "March" "1994" "17:42:36" "GMT" "Don Wells" "dwells at nrao.edu" "<DWELLS.94Mar22124238 at fits.cv.nrao.edu>" "55" "Re: 3D fits files ?" "^From:" nil nil "3" "1994032217:42:36" "3D fits files ?" (number " " mark "     Don Wells         Mar 22   55/2312  " thread-indent "\"Re: 3D fits files ?\"\n") "<2mm9jj$kkq at menudo.uh.edu>"]
	nil)
Received: by fits.cv.nrao.edu (4.1/DDN-DLB/1.5)
	id AA02979; Tue, 22 Mar 94 12:46:30 EST
Return-Path: <news at saips.CV.NRAO.EDU>
Message-Id: <DWELLS.94Mar22124238 at fits.cv.nrao.edu>
Organization: nrao
Path: saips.cv.nrao.edu!news.cv.nrao.edu!dwells
References: <2mm9jj$kkq at menudo.uh.edu>
From: dwells at nrao.edu (Don Wells)
Sender: fitsbits-request at fits.CV.NRAO.EDU
To: fitsbits at fits.CV.NRAO.EDU
Subject: Re: 3D fits files ?
Date: Tue, 22 Mar 1994 17:42:36 GMT

In article <2mm9jj$kkq at menudo.uh.edu> svec5 at menudo.uh.edu (T.C. Zhao) writes:
     ".. ftp site.. fits files with NAXIS=3.."

    ftp://fits.cv.nrao.edu/fits/sampledata/cubes/ngc6503.fits

This is an 8.2 Mbyte file. Some relevant header lines are:

BITPIX  =                  -32 /
NAXIS   =                    4 /
NAXIS1  =                  256 /
NAXIS2  =                  256 /
NAXIS3  =                   31 /
NAXIS4  =                    1 /
OBJECT  = 'NGC6503 '
OBSERVER= 'WELL    '
DATE-OBS= '12/04/83'
DATE-MAP= '20/09/84'
BUNIT   = 'JY/BEAM '
DATAMAX =      2.953770570E-02 /MAX PIXEL VALUE
DATAMIN =     -1.367387734E-02 /MIN PIXEL VALUE
CTYPE1  = 'RA---SIN'
CTYPE2  = 'DEC--SIN'
CTYPE3  = 'FELO-HEL'
CRVAL3  =    2.60000000000E+04 /
CDELT3  =     -1.030761914E+04 /
CRPIX3  =      1.600000000E+01 /
CROTA3  =      0.000000000E+00 /
CTYPE4  = 'STOKES  '
CRVAL4  =    1.00000000000E+00 /

This 21cm cube is a *textbook case* of a spiral galaxy rotation curve.
NGC 6503 is a normal spiral galaxy with morphological type SA(s)cd.
The VLA observation was by me 11 years ago. It used 21 antennas in C
array, 32 channels with 48 KHz (10 km/s) spacing. Exposure time on the
galaxy was about 2 hours. The VLA has much better receivers today, and
27 antennas could now be used for such a spectral line observation.

The pixels are 32-bit IEEE floating point, ranging from -13 to +29
milli-Janskys per beam. This is a 4-dimensional matrix (NAXIS=4), but
the 4th axis (of type 'STOKES') has only one pixel.  The third axis
has 31 pixels, and they are expressed in heliocentric velocity evenly
spaced in frequency (that is what 'FELO-HEL' means). Pixel 16 (the
central channel of the spectra) has a velocity of 26 km/s (26000 m/s),
and the channel spacing is -10.3 km/s. The 26 km/s was the book value
for the redshift of 6503 ten years ago; NED (IPAC's extragalactic
database) says the current book value is "Helio. radial vel. : 44 +/-
3 km/s".

This is a fun piece of data, suitable for movie-loops and 3-D
visualization systems like AVS. Enjoy!
--
  Donald C. Wells         Associate Scientist         dwells at nrao.edu
                    http://fits.cv.nrao.edu/~dwells
  National Radio Astronomy Observatory                +1-804-296-0277
  520 Edgemont Road,   Charlottesville, Virginia       22903-2475 USA

From fitsbits-request  Fri Mar 25 17:45:54 1994
Status: RO
X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil]
	["157" "" "25" "March" "1994" "22:05:36" "GMT" "Greg F. Walz Chojnacki" "gwc at alpha1.csd.uwm.edu" "<2mvn7gINNp91 at uwm.edu>" "7" "FITS viewer for Mac?" "^From:" nil nil "3" "1994032522:05:36" "FITS viewer for Mac?" (number " " mark "     Greg F. Walz Choj Mar 25    7/157   " thread-indent "\"FITS viewer for Mac?\"\n") nil]
	nil)
Received: by fits.cv.nrao.edu (4.1/DDN-DLB/1.5)
	id AA13192; Fri, 25 Mar 94 17:45:54 EST
Return-Path: <news at saips.CV.NRAO.EDU>
Message-Id: <2mvn7gINNp91 at uwm.edu>
Organization: Computing Services Division, U of Wis - Milwaukee
Path: saips.cv.nrao.edu!hearst.acc.Virginia.EDU!darwin.sura.net!rsg1.er.usgs.gov!stc06r.CTD.ORNL.GOV!fnnews.fnal.gov!uwm.edu!alpha1.csd.uwm.edu!gwc
From: gwc at alpha1.csd.uwm.edu (Greg F Walz Chojnacki)
Sender: fitsbits-request at fits.CV.NRAO.EDU
To: fitsbits at fits.CV.NRAO.EDU
Subject: FITS viewer for Mac?
Date: 25 Mar 1994 22:05:36 GMT

The subject line says it all: I'm looking for a utility for viewing -- and,
ideally, manipulating, at least rudimentarily, FITS image files.

Thanks.

Greg

From fitsbits-request  Tue Mar 29 11:05:03 1994
Status: RO
X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil]
	["2409" "Tue" "29" "March" "1994" "16:03:33" "GMT" "Don Wells" "dwells at nrao.edu" "<DWELLS.94Mar29110334 at fits.cv.nrao.edu>" "47" "Happy Birthday, FITS!" "^From:" nil nil "3" "1994032916:03:33" "Happy Birthday, FITS!" (number " " mark "     Don Wells         Mar 29   47/2409  " thread-indent "\"Happy Birthday, FITS!\"\n") nil]
	nil)
Received: by fits.cv.nrao.edu (4.1/DDN-DLB/1.5)
	id AA00613; Tue, 29 Mar 94 11:05:03 EST
Return-Path: <news at saips.CV.NRAO.EDU>
Message-Id: <DWELLS.94Mar29110334 at fits.cv.nrao.edu>
Organization: nrao
Path: saips.cv.nrao.edu!news.cv.nrao.edu!dwells
From: dwells at nrao.edu (Don Wells)
Sender: fitsbits-request at fits.CV.NRAO.EDU
To: fitsbits at fits.CV.NRAO.EDU
Subject: Happy Birthday, FITS!
Date: Tue, 29 Mar 1994 16:03:33 GMT

My copy of the (handwritten) Basic FITS Agreement is dated 
29 March 1979, so FITS is 15 years old today. Happy Birthday, FITS!

Sometime in the mid-80s NRAO-Greenbank wanted to clean out their stock
of old reprints, and they sent their remaining original reprints for
the Basic-FITS paper to Charlottesville; Eric said I should keep them.

As a birthday present for FITS, I offer these original reprints to the
FITS community. I will mail them to individuals in the order in which
I receive Email requests, until the stock is exhausted.

I have 43 copies of the June 1981 A&A Suppl. Basic-FITS paper.

I also have 52 copies of the 1981 Random-Groups paper, and will
include them with the Basic-FITS reprints.

Finally, the reprint collection includes 42 copies of a little-known
paper "The FITS tape formats: flexible image transport systems", by
Greisen, Wells and Harten, pp.298-300 in SPIE Vol. 264, "Applications
of Digital Image Processing to Astronomy". This volume is the
proceedings of an IP meeting held at JPL/Caltech in August 1980, so
this paper appeared in print before the Basic-FITS and Random-Groups
papers. It is a 3-page overview of the early FITS agreements which is
notable for its Figure 2 "A schematic illustration of the most general
structure used for the binary data". This figure is a wonderful 3-D
illustration of how random-groups data structures cross over logical
record boundaries. The concept of this illustration is also fully
applicable to binary tables data structures, of course, and I wish
that this figure could somehow be recycled and appear in FITS
document(s) in the 90s (perhaps in the binary tables paper?).

At that August 1980 IP meeting at JPL there was a special
birds-of-a-feather session on the problem of coordinating development
of astronomical software. The formation of the AAS Working Group for
Astronomical Software [WGAS] was the primary result of that session.

                -*-

Two years ago, my birthday present for FITS was a long report on the
developments leading up to FITS, and on the early history of FITS.
See ftp://fits.cv.nrao.edu/fits/documents/overviews/history.news.
--
  Donald C. Wells         Associate Scientist         dwells at nrao.edu
                    http://fits.cv.nrao.edu/~dwells
  National Radio Astronomy Observatory                +1-804-296-0277
  520 Edgemont Road,   Charlottesville, Virginia       22903-2475 USA

From fitsbits-request  Thu Mar 31 16:25:40 1994
Status: RO
X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil]
	["1366" "Thu" "31" "March" "1994" "16:25:27" "EST" "William Pence" "pence at tetra.gsfc.nasa.gov" "<9403312125.AA27522 at tetra.Gsfc.NASA.Gov>" "46" "FITSIO Questionnaire" "^From:" nil nil "3" "1994033121:25:27" "FITSIO Questionnaire" (number " " mark "     William Pence     Mar 31   46/1366  " thread-indent "\"FITSIO Questionnaire\"\n") nil]
	nil)
Received: by fits.cv.nrao.edu (4.1/DDN-DLB/1.5)
	id AA06969; Thu, 31 Mar 94 16:25:40 EST
Return-Path: <pence at tetra.gsfc.nasa.gov>
Message-Id: <9403312125.AA27522 at tetra.Gsfc.NASA.Gov>
From: pence at tetra.gsfc.nasa.gov (William Pence)
Sender: fitsbits-request at fits.CV.NRAO.EDU
To: fitsbits at fits.CV.NRAO.EDU
Cc: pence at tetra.gsfc.nasa.gov
Subject: FITSIO Questionnaire
Date: Thu, 31 Mar 94 16:25:27 EST

To users of the FITSIO subroutine library:

In order to gauge the extent that the FITSIO subroutine library is
being used, and to help justify its ongoing support, I would be
grateful if any users of the FITSIO software would please fill out and
return to me the following short questionnaire (do not send it back to
this newsgroup).  I am interested in hearing from individual
programmers as well as larger projects that have used FITSIO,
especially other organizations within NASA.

Thank you in advance for your support,
Bill Pence
NASA/GSFC

----- Cut Here -----------------------------------------------

           FITSIO Software Questionnaire

       (please return to pence at tetra.gsfc.nasa.gov)

1.  Name of individual or project that used FITSIO:

2.  Institution address:

3.  How useful was FITSIO for this project (check one)

  _____ a) not very useful; could easily cope without using FITSIO

  _____ b) moderately useful; FITSIO made it significantly easier or
           less costly to complete the project.

  _____ c) very useful;  the project would have been difficult or much
           more costly to complete without using FITSIO

4. (optional) Briefly describe your project and the role played 
   by FITSIO:

5. (optional) Suggestions for making FITSIO more useful:

----- End of Questionnaire ----------------------------------------

From fitsbits-request  Thu Mar 31 17:16:13 1994
Status: RO
X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil]
	["2054" "" "31" "March" "1994" "16:52" "EDT" "BARRY M. SCHLESINGER" "bschlesinger at nssdca.gsfc.nasa.gov" "<31MAR199416522547 at nssdca.gsfc.nasa.gov>" "40" "Re: No ST-6 FITS images ?" "^From:" nil nil "3" "1994033120:52:00" "No ST-6 FITS images ?" (number " " mark "     BARRY M. SCHLESIN Mar 31   40/2054  " thread-indent "\"Re: No ST-6 FITS images ?\"\n") "<94088.142150K3032E0 at ALIJKU11.BITNET>"]
	nil)
Received: by fits.cv.nrao.edu (4.1/DDN-DLB/1.5)
	id AA07040; Thu, 31 Mar 94 17:16:13 EST
Return-Path: <news at saips.CV.NRAO.EDU>
Message-Id: <31MAR199416522547 at nssdca.gsfc.nasa.gov>
Organization: NASA - Goddard Space Flight Center
Path: saips.cv.nrao.edu!hearst.acc.Virginia.EDU!darwin.sura.net!howland.reston.ans.net!cs.utexas.edu!swrinde!elroy.jpl.nasa.gov!ames!newsfeed.gsfc.nasa.gov!nssdca.gsfc.nasa.gov!bschlesinger
References: <94088.142150K3032E0 at ALIJKU11.BITNET>
From: bschlesinger at nssdca.gsfc.nasa.gov (BARRY M. SCHLESINGER)
Sender: fitsbits-request at fits.CV.NRAO.EDU
To: fitsbits at fits.CV.NRAO.EDU
Subject: Re: No ST-6 FITS images ?
Date: 31 Mar 1994 16:52 EDT

In article <94088.142150K3032E0 at ALIJKU11.BITNET>, Herbert Raab <K3032E0 at ALIJKU11.BITNET> writes...
>I once read a posting saying that FITS images written by the software of
>the ST-6 CCD camera do not fullfill the FITS standard.... 
> 
>What's wrong with the files written by the ST-6 software ?
> 
	
	Funny that this question should come up at this time.  They
use unsigned 16-bit integers, which, as we have been discussing here 
in the past month, FITS does not support. 

	While we are on the subject....  My suggestion for a new
extension type to hold unsigned integers in binary tables was not
intended as a permanent solution.  The idea is that such an extension
type would be used to hold the new forms during the process of
development.  Should the community consensus be to approve the new
data types, after approval they would be allowed in an extension with
type name BINTABLE. 

	About the calls for fast action....  I have added a paragraph on 
the procedures by which FITS evolves to the monthly basics and 
information posting.  (If it's gone at your site, it's available from
ftp://nssdca.gsfc.nasa.gov/fits/basics_info.txt)  Note that the 
process is deliberate.  This choice is no accident.  One of the
criticisms of FITS is that it has some awkward features.  Such
features arise largely because of the rule that no change can be
made that would invalidate a valid FITS file.  The presence of this
rule means that material can be cast into FITS format without fear
that at some time in the future a change in the rules will require
conversion of an extensive data library.  Providing users such
confidence is worth the cost of occasionally having to make constructs
somewhat more complicated than one would like.  However, because, as a
result, any extension to FITS will limit what can be done in the
future, it is essential that such additions be thought through very
carefully before endorsement, so that the future flexibility of FITS
not be unnecessarily compromised. 

				Barry Schlesinger
				NSSDC/NOST FITS Support Office	

From fitsbits-request  Thu Mar 31 20:06:59 1994
Status: RO
X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil]
	["1055" "Thu" "31" "March" "1994" "14:16:12" "GMT" "Rob Verschoor" "rverscho at isosa2.estec.esa.nl" "<CnJ8z0.5sA at isosa2.estec.esa.nl>" "29" "Re: FITS viewer for Mac?" "^From:" nil nil "3" "1994033114:16:12" "FITS viewer for Mac?" (number " " mark "     Rob Verschoor     Mar 31   29/1055  " thread-indent "\"Re: FITS viewer for Mac?\"\n") "<2mvn7gINNp91 at uwm.edu>"]
	nil)
Received: by fits.cv.nrao.edu (4.1/DDN-DLB/1.5)
	id AA07152; Thu, 31 Mar 94 20:06:59 EST
Return-Path: <news at saips.CV.NRAO.EDU>
Message-Id: <CnJ8z0.5sA at isosa2.estec.esa.nl>
Organization: ESA/Estec/SAI, Noordwijk, The Netherlands
Path: saips.cv.nrao.edu!hearst.acc.Virginia.EDU!darwin.sura.net!howland.reston.ans.net!EU.net!sun4nl!estwmz.wm.estec.esa.nl!isosa2.estec.esa.nl!isosa2!rverscho
References: <2mvn7gINNp91 at uwm.edu>
From: rverscho at isosa2.estec.esa.nl (Rob Verschoor)
Sender: fitsbits-request at fits.CV.NRAO.EDU
To: fitsbits at fits.CV.NRAO.EDU
Subject: Re: FITS viewer for Mac?
Date: Thu, 31 Mar 1994 14:16:12 GMT

In sci.astro.fits you write:

>The subject line says it all: I'm looking for a utility for viewing -- and,
>ideally, manipulating, at least rudimentarily, FITS image files.

There's no such thing as a 'FITS' viewer. FITS is a convention for formatting 
any kind of data in a transportable format. A FITS viewer would therefore be 
about as meaningful as an 'ISAM' file viewer: not at all. FITS data can be 
anything : spectra, images, whatever. 
What you probably want are packages like IDL or MIDAS (better take IDL) 
which provide facilities for processing spectra etc. and are also 
capable of reading/writing data in FITS format.
If you want to access FITS data yourself, you could use NASA's FITSIO package.
However, this is in Fortran and although various platforms are supported, 
Mac isn't one of them (but maybe this has changed). You get it by 
anonymous ftp to tetra.gsfc.nasa.gov (pub/fitsio3). Good luck...

Rob Verschoor

rverscho at isosa2.estec.esa.nl

5 ........	White (o) to play...
4 ..oooo..
3 oo.**o..
2 **..*o..
1 ..***o..
  abcdefgh