**Using the Java FITS utilities**.

This document describes version 0.91 of the nom.tam.fits class library
for reading and writing FITS files in Java. It assumes some familiarity
with both FITS and Java. For a discussion of FITS see documentation at
[http://fits.gsfc.nasa.gov](http://fits.gsfc.nasa.gov/). FITS, the
Flexible Image Transport System, is a file format defined to facilitate
the exchange of scientific information. This class library provides
facilities to convert FITS information to forms usable directly within
Java programs, and to write information generated in a program to a FITS
file.

The underlying philosophy for this library is to make simple things easy
but leave hard things possible. The library encompasses quite a few
classes, but users will typically need to use only a small fraction of
the public interface of the system.

This document provides a quick overview of the packages and classes in
the system and then discusses how to use the library using a series of
simple examples. Special topics are discussed at relevant points. A list
of major changes to the public interface since the last major release is
appended.

# Overview of the System

The FITS classes comprise the classes in the nom.tam.fits package and
make extensive use of the nom.tam.util package. The nom.tam.image
package provides support for dynamic image subsetting.

The current release of the package requires Java 1.2. In particular the
Header class uses Collections

and there may be occasional dependencies upon methods that were defined
in Java 1.2.

**nom.tam.util:**

**BufferedDataInputStream**

**BufferedDataOutputStream**

**BufferedFile**

**ArrayDataInput**

**ArrayDataOutput**

**RandomAccess**

> These classes are used to provide efficient access to large amounts of
> data. Basically they allow arrays to be read and written efficiently.
> The Stream classes provide input and output for streams, while the
> BufferedFile class provides efficient access for uncompressed, local
> data using an underlying RandomAccessFile. These routines are typical
> 5-100 times faster than the comparable standard Java classes. The
> standard classes are slowed by frequent synchronization and the many
> method invocations required to read large arrays.
>
> The ArrayDataInput and ArrayDataOutput classes are interfaces that are
> implemented as appropriate by the BufferedXXX classes. The
> RandomAccess interface extends ArrayDataInput. It is implemented only
> by BufferedFile currently.
>
> These classes are general utilities and can be used completely
> independently of the FITS library.

**ByteParser**

**ByteFormatter**

> These two classes provide more efficient translations between ASCII
> and binary representations of numbers than the standard FITS classes.
> Note that for real numbers these classes may suffer small round-off
> errors compared to the standard classes.
>
> These classes may be useful outside of the FITS context but so far are
> used only in FITS ASCII table classes.

**ColumnTable**

**DataTable**

> These classes are used within FITS binary tables. The ColumnTable
> class provides a mechanism for efficient access to non-homogeneous
> data. They also allow a Binary table to be created using relatively
> few objects (essentially one object per table) rather than one object
> per entry. Both of these dramatically enhance the useability of binary
> tables. The DataTable interface defines how a ColumnTable is to be
> accessed but it not used in the FITS libraries. The ColumnTable should
> be usable outside of the FITS library whenever a table of
> heterogeneous data is to be read. Homogeneous data (i.e., data
> composed of a single primitive type) is more likely to be easily read
> with classes that implement the ArrayDataXPut interfaces.

#### ArrayFuncs

> The ArrayFuncs class defines a large number of static utility
> functions for manipulating arrays, especially primitive arrays. Most
> handle multi-dimensional arrays transparently. Facilities include:

-   Generating deep clones of arrays.

-   Copying an array to an array of another type.

-   Determining the total size of an array.

-   Determining the base type of an array.

-   Converting a multi-dimensional array to one-dimension (flattening).

-   Converting a one-dimensional array to multiple dimensions (curling).

-   Examining an array.

-   Extracting the shape of an array.

#### HashedList

#### Cursor

> The HashedList is a Collection defined to support FITS headers.
> Basically it provides support for a linked list where elements may
> optionally have keys. The inner class HashedList.HashedListIterator
> implements the Cursor interface and is extremely helpful in
> manipulating FITS headers. The Cursor interface extends the
> java.util.Iterator interface but allows insertions and keyed access.
> The Header.iterator() method returns a Cursor that the user can use to
> view the header.

**nom.tam.image:**

**ImageTiler**

> The ImageTiler class allows users to extract subimages from a FITS
> primary image or image extension.

**nom.tam.fits:**

**Fits**

The FITS class is primarily concerned with establishing a connection
between the FITS object and outside world. A wide variety of
constructors allow the user to read existing FITS data. A null FITS
object can be created to which HDU's can later be attached.

A tiny note on capitalization: In this FITS libraries, acronyms that are
pronounced as words, e.g., FITS and ASCII, are treated as normal words
in class and variable names. Acronyms that are spelled out, e.g., HDU,
are always (I hope!) capitalized.

#### XxxHDU/XxxData

Each of the types of FITS data uses a pair of classes XxxHDU and XxxData
(or XxxTable). The HDU class provides the links between the header and
data sections of an HDU while the Data class provides the detailed
analysis and access to the underlying FITS data. Each FITS type also has
an associated data kernel, a non-FITS structure in which data is
actually held. For binary tables this is a ColumnTable but it is some
variety of Java array for all other types.

#### ImageHDU/ImageData

> These classes now include the functionality of the old PrimaryHDU and
> PrimaryData classes. Users can either retrieve FITS data as a
> multi-dimensional array or use an ImageTiler to get sub-images of the
> array as a one-dimensional array. Reading of image data is typically
> deferred until the user requests data. **Extension:** Java longs are
> supported as images with BITPIX=64.

#### RandomGroupsHDU/RandomGroupsData

These classes support FITS random groups. Random groups are permitted in
Image extensions as well as in the primary array. Random groups data is
supported as a Object\[nrow\]\[2\] array. The first element for each
array is the object parameter information -- which may have 0 length.
The second element is the data array. **Extensions**: Random groups
support BITPIX=64 longs. Random groups are supported in Image
extensions.

#### AsciiTableHDU/AsciiTable

> These classes support FITS ASCII tables. The ASCII table kernel is an
> Object\[\] array. Each element of this array is a primitive double,
> float, int, or long array, or a String array. The constituent arrays
> must all have the same size. Users can construct an ASCII table
> dynamically by adding columns starting from a null array or by using
> the appropriate constructors. Note that the classes currently ignore
> the number of decimal places specified in the TFORM entry for real
> numbers. Only the length information is used in formatting data. Null
> fields are fully supported.
>
> ASCII table data is not normally read until the user requests it.
> Users may request data by element, row, or column. The last of these
> will read in the entire table, but the first two will read only the
> requested data. The AsciiTable.getData() (or getKernel) method can
> also be used to ensure that all data is read.
>
> All ASCII tables can be represented as binary tables and users can
> enable or disable binary tables by calling the setUseAsciiTables
> method in FitsFactory. If ASCII tables are enabled they will be used
> where possible.

Note that ASCII tables support long integers but this is not an
extension of the FITS standard. Indeed FITS ASCII tables can in
principal store real and floating point numbers of arbitrary length and
precision which may not be representable using any of the standard Java
types. This is unlikely to be a problem in practice though it is not
inconceivable that there are FITS files with 8 byte integer or 16 byte
real data encoded in ASCII tables.

**BinaryTableHDU/BinaryTable/FitsHeap**

These classes support FITS binary tables. Variable length columns are
supported. Variable length column elements are returned with appropriate
lengths and may be returned as zero-length arrays.

> The data kernel is a ColumnTable object and row and element reads are
> possible without requiring the kernel to be instantiated.
> **Extension:** Long integers are supported in binary tables using the
> format character (in TFORM) 'K'.
>
> **UnknownHDU/UnknownData**
>
> These classes support FITS data where the internal structure of the
> FITS information is not known or currently supported. Data is stored
> internally as a byte array. You can actually create an HDU
>
> of this type to buffer conglomerations of primitive data types. Most
> commonly these type can be used to read standard formats in an
> installation of the FITS library where not all formats are supported.

#### FitsFactory

The FitsFactory class is used to find the appropriate FITS type. It
allows users to create FITS data elements given a Header, or an HDU
given a data element. When adding a new type of data to be handled in
the FITS library, only the FitsFactory and the classes directly
supporting the new type should need to be modified. The FITS classes now
support all the accepted protocols so further extensions may be rare.
Users can get most of the functionality of the FitsFactory class using
convenience methods in the Fits class.

#### FitsUtil

> This class comprises a few utilities that are needed in various
> elements of the FITS classes. Users should not typically need to
> access this class directly.

## FitsDate

This class provides for translations between FITS and Java
representations of dates. Both the old and new FITS date formats may be
read.

**Deferred Input.**

Most FITS classes support deferred input for FITS data. If a FITS HDU is
read from a non-compressed local file, then the header is read but the
data section is skipped until the user requests information from it. If
the user requests an entire image or table it will be read, but users
may choose to read only sections of the data as appropriate for the
particular type, e.g., a subset of an image or a row of a table. In this
case the entire data element need never fully present in memory. Once
the entire data section is read into memory, operations to read in
sections of the data are still supported but work from the in-memory
version rather than from the input file.

While deferred input should normally be invisible to users, it is
possible to cause problems if the user mangles the FITS input stream
between the time the HDU is initially scanned and when the user
eventually reads the file. Note also that users must provide any
synchronization needed to manage multiple accesses to a given FITS
resource.

**Rewriting and rereading data.**

All FITS types support re-writing if the data is being read from an
uncompressed local file. The system attempts to assure that the size of
an HDU element has not changed when a re-write is requested. Note that
elements are always a multiple of 2880 bytes, so there is some
flexibility here.

## Examples

The examples below sketch out how a user might perform certain
functions. For maximum clarity no error checking code is shown. Examples
of most of the useful calls in the FITS library are found in the
nom.tam.fits.test package in the \*Tester classes.

**Read the primary image:**

> Fits f = new Fits("filename");
>
> ImageHDU h= (ImageHDU) f.readHDU();
>
> float\[\]\[\] img = (float\[\]\[\]) h.getKernel();

While this is simple enough, note the ugly coercions required to get
data. I have not been able to find any way of getting around these. Also
note that this code assume you know the type of the data. If not you
might want to use ArrayFuncs to parse the object returned.

**No scaling:**

One important thing to note about the FITS classes is that they never
automatically scale data for you. You get the data exactly as it was
stored in the FITS classes.

**Get a subset without reading the entire image:**

> Fits f = new Fits("filename");
>
> ImageHDU h= (ImageHDU) f.readHDU();
>
> ImageTiler t = h.getTiler();
>
> float\[\] img= tiler.getTile(new int\[\]{10,10}, new int{30,30});

This gets a 30x30 tile with lower left corner (in FITS directions) at
10,10. Note the use of the immediate array declarations. They are ugly
but convenient. Remember that one can declare arrays like:

> new int\[\]{x,y}

where x and y are variables. You can get as many subsets as you like. If
you ever call h.getKernel() the image will be read into memory and then
subsets will be derived from the in-memory region. This might be nice if
you want to do something like an animation.

If you need to get a lot of tiles it may be inefficient to create a new
array for each tile. The getTile method is overloaded to allow the user
to supply the input array. If this version is called, then they user may
request a tile which is not fully (or even partially) contained within
the image. Pixels that are not available will be left unchanged.

**Create a FITS file from an image:**

> double\[\]\[\] x = ....
>
> Fits f = new Fits();
>
> BasicHDU h = FitsFactory.HDUFactory(x);
>
> f.addHDU(h);
>
> BufferedDataOutputStream s = new
>
> BufferedDataOutputStream("OutputFile");
>
> f.write(s);

There are also makeHDU methods in the Fits to bypass calling the
FitsFactory directly. I.e.,

BasicHDU h = Fits.makeHDU(x);

These create an HDU but do not add it to a Fits object.

#### Read an entire FITS file and get a summary of its contents

> Fits f = new Fits("Filename");
>
> BasicHDU\[\] hdus = f.read();
>
> for (int i=0; i\<hdus.length; i += 1) {
>
> hdus\[i\].info();
>
> }

Note that this won't do anything if there's a problem at the end of the
file because it will crash before it starts writing. It's a little safer
to say

> do {
>
> BasicHDU h = f.readHDU();
>
> if (h != null) {
>
> h.info();
>
> }
>
> } while (h != null) ;

Then you'll get information on any HDU's at the beginning of the file
even if the end is corrupt.

Note, when you call readHDU the Fits object remembers the HDU it read,
so that you can go back to it if you want to.

E.g., after doing the above we could have

> BasicHDU h = f.getHDU(0);

to get the primary array.

**Read random groups data:**

Fits f = new Fits("RandomGroupsFile");

RandomGroupsHDU h = (RandomGroupsHDU) f.readHDU();

Object\[\]\[\] data = (Object\[\]\[\])h.getData().getData();

for (int i=0; i\<data.length; i += 1) {

int\[\] params = (int\[\]) data\[i\]\[0\];

int\[\]\[\] img = (int\[\]\[\]) data\[i\]\[1\];

... Process a group ...

}

More ugly casting I'm afraid. Remember that you need to know the type to
cast to. This is the cost of a statically typed language. You can create
a random groups in the same way as for an image. Just make sure that
your data is an appropriately formatted Object\[\]\[\] array.

Currently random groups don't support deferred input.

## Manipulate the FITS header

The recommended approach for this much changed in this release. We can
get the Header from an HDU:

> Header hdr = someHDU.getHeader();

There are methods in Header to access it, e.g.:

> if (hdr.getIntValue("TLMIN3")) {
>
> ...
>
> }

and you can add a card to the Header with

> hdr.addValue("TLMIN4", 36, "This is a comment");

Note that this will replace the old value of TLMIN4 if it existed.

The new recommended approach to managing the Header is to use a Cursor
to manipulate a new Collection, a HashedList. A HashedList is just a
linked list where some of the elements can be accessed using a hash. The
FITS header is stored in a HashedList. A Cursor is a kind of
super-Iterator to

manipulate this list.

Here's how it works:

> Cursor iter = hdr.iterator();
>
> while (iter.hasNext()) {
>
> HeaderCard hc = (HeaderCard) iter.next();
>
> String key = hc.getKey();
>
> ...
>
> }

A Cursor can be moved to any point in the Header and it can also be used
to add cards to the header. E.g., suppose we want add TCX1, TCX2, and
TCX3 keywords after the TCX keyword, and delete any comments we find
immediately after the TCX keyword.

> iter.setKey("TCX"); // Move to just before this keyword
>
> if (iter.hasNext()) { // Would be false if TCX wasn't in header
>
> iter.next(); // Now after TCX
>
> iter.add("TCX1",
>
> new HeaderCard("TCX1", someValue, someComment);
>
> iter.add("TCX2",
>
> new HeaderCard("TCX2", someValue, someComment);
>
> iter.add("TCX3",
>
> new HeaderCard("TCX3", someValue, someComment);
>
> while (iter.hasNext()) {
>
> String key = iter.next().getKey(); // Get the key
>
> if (key != null && key == "COMMENT") {
>
> iter.remove(); // Remove the comment
>
> } else {
>
> break;
>
> }
>
> }
>
> }

This is a little longer than our previous examples, but let's see what
we did. First we moved the cursor to point just before a particular
keyword. If the keyword were not in the header we'd point to the end of
the header. Now we wanted to add cards after this keyword, so we got it
and then added three new cards.

Note how were able to add as many keywords as we wanted. Also, note that
the new cards are added just before the current position of the cursor,
so the add calls don't affect what is returned by iter.next(). Then we
checked the subsequent cards to see if there were immediately following
comments. If so we deleted them. We can only delete one entry per call
to next().

A few things to note. There are two kinds of keys here. HashedList keys
are specified by the user as the first argument in the two argument add
call. They'll also get set when we read key=value cards in an existing
header. These keys must be unique. Each HeaderCard can also have a key.
For a key=value card this should normally be the same as the HashedList
key, but for other cards this might be 'COMMENT' or 'HISTORY'. In
principle one could create HashedList keys to COMMENT or HISTORY cards,
but the user needs to be careful not to create duplicate keys. If we try
to add a keyed entry to the HashedList, it will delete an existing
element with the same key0.

One can have many Cursors on the same Header, which may be useful if
different objects are looking at different parts of the header, but the
user can get in trouble with incautious deletions.

When an HDU is created from a data object, all required structural
keywords will be inserted into the Header. A user modifies those at
their own risk. When a Header is written the initial keywords are
checked to see if they are legal, but this check is fairly minimal.

The Header.positionAfterIndex method is useful when you want to add
information about something in the header that takes an index (e.g.,
TLMIN, or CRVAL).

**HIERARCH key values:**

These are not supported now, but here's the plan for how that get's
added in...

A class HierarchCard extends the HeaderCard. To add a HierarchCard one
just does

> iter.addKey(key, new HierarchCard(...));

The only problem is how to ensure when a header is read, the HIERARCH
cards are properly handled. My current thought is that if you want
HIERARCH cards to be recognized, then there will be a static method in
HierarchCard which will process and existing header and replace
appropriate HeaderCards with HierarchCards. E.g.,

> HierarchCard.updateHeader(anHDU.getHeader());

This makes it very easy to support the convention but requires a user to
recognize that they are invoking an extension to standard FITS. One
bonus to this approach is that I believe it would be quite feasible to
support real hierarchies. E.g., suppose with have cards of the form:

> HIERARCH MISSION INSTRUMENT SUBSYSTEM someValue

and we want to find all the keywords associated with a given instrument.
One could design this such that a call like:

> HierarchCard\[\] allCards = HierarchCard.getMatch(header,
>
> "MISSION.INSTRUMENT.\*");

**Read an image line-by-line:**

Here's something a little funkier where we use the lower level methods
in the classes. We'll read in an image one-line at a time. You can judge
whether

it is a good thing to support a style like this.

> BufferedFile bf = new BufferedFile("Input.fits");
>
> Header h = Header.readHeader(bf);
>
> int naxes = h.getIntValue("NAXIS");
>
> int lastAxis = h.getIntValue("NAXIS"+naxes);
>
> h.addValue("NAXIS", naxes-1);
>
> ImageData d = ImageData.manufactureData(h);
>
> float\[\] line = (float\[\]) d.getData();
>
> for (int i=0; i\<lastAxis; i += 1) {
>
> bf.read(line);
>
> ... Process a line.
>
> }

Here's what we did: First we opened the file directly as a BufferedFile
rather than using the Fits class. We read only the first header. Then
things got devious. We modified the header so that it described
something with one less dimension than originally. E.g., if the header
originally had described a 2-D real array it now described only one line
of that array. Using this modified header we got a data object so that
we could get an array of the appropriate dimensions and type to read the
data. Then we read the image data explicitly using the methods of
BufferedFile.

Is there any reason for this subterfuge? Probably not but is illustrates
a little of the inner workings of the FITS classes.

**Read a compressed file:**

If the file is Gzip compressed and ends in '.gz', just use the file name
and this will be handled automatically. If not you may want to do
something like:

> File fl = new File("filename");
>
> Fits f = new Fits(fl, true);

The second argument indicates that the File is compressed.

If you have a FITS file compressed using some other scheme then you'll
need an inflater. E.g.,

Fits f = new Fits(SomeInflaterStream("SomeCompressedFile"));

Note that deferred reads and re-writes are not supported for compressed
files.

**Read a URL:**

Just enter the full URL for HTTP protocol URLs. You can also use the URL
constructors.

> Fits f = new Fits("<http://xyz.edu/somedata.fits>");

or

> URL u = new URL("<http://xyz.edu/somedcompressedFitsData.ff>");
>
> Fits f = new Fits(u, true);

**Read an ASCII table:**

At its simplest reading an ASCII table almost as easy as an image. One
big difference though is that tables can never be the primary data for a
FITS file.

The data for an ASCII table is returned as a set of arrays, one for each
column in the table.

> Fits f = new Fits("fileWithAsciiTable");
>
> BasicHDU\[\] hdus = f.read();
>
> Object\[\] cols = (Object\[\]) hdus\[1\].getKernel();
>
> float\[\] exposures = (float\[\]) cols\[0\];
>
> int\[\] counts = (int\[\]) cols\[1\];
>
> String\[\] pi_name = (String\[\]) cols\[2\];
>
> ... these arrays have the same dimensions ...

**Read pieces of an ASCII table:**

You don't have to read the entire table. After the first two lines we
might have:

> AsciiTable a = (AsciiTable) hdus\[1\].getData();
>
> Object\[\] firstRow = a.getRow(0);

At this point firstRow is just like cols above, except that all of the
arrays have only a single element in the, the elements for the first
row. It's not elegant, but you still have to do something like:

> float\[\] fa = (float\[\])firstRow\[0\];
>
> float expos = fa\[0\];

One vital thing to note: When referring to FITS rows and columns, the
FITS library uses the 0-based Java indices for these, not 1-based FITS
indices. I.e., the column that has 'TFORM1' is column 0. This is
confusing. Alas, the alternatives seem just as bad.

You can also just get a single element:

> String\[\] sa = (String\[\]) a.getElement(29,2);

gets the PI for the 30'th row.

And of course one can get a column:

> int\[\] counts = (int\[\]) a.getColumn(1);

Reading rows and elements does not require reading the entire extension.
AsciiTables support deferred reads. However a getColumn() call will read
the entire table into memory. The getRow and getElement calls will still
work -- they'll be even faster -- but the entire table will be taking up
space in memory. You

can always force the data for any HDU type to be read into memory with

> a.getData();

If you want to use the getRow or getElement methods to read the entire
table, it's probably faster to precede them with a getData() unless the
table is very large.

Large ASCII tables can take a long time to read but fortunately they are
relatively rare.

**Writing an ASCII table:**

If you have a set of simple String, int, long, float and double arrays
it's easy to write an ASCII table.

Object\[\] baseObj = new Object\[\] {arr1, arr2, arr3, ...};

Fits f = new Fits();

FitsFactory.setUseAsciiTables(true);

Fits.addHDU(Fits.makeHDU(baseObj));

BufferedFile bf = new BufferedFile("outputfile", "rw");

bf.write(bf);

bf.flush();

bf.close();

Note that we told the system that we wanted to have ASCII tables. That's
the current default, but it never hurts to make sure! Also note that we
added only the ASCII table to the Fits object. The Fits object inserted
a dummy primary HDU for us automatically.

**Modifying an ASCII table:**

There are methods corresponding to the getRow, getColumn and getElement
routines to allow the user to modify the contents of existing rows and
columns. A user can also build a table column by column.

> AsciiTable d = new AsciiTable();
>
> d.addColumn(anArray);
>
> d.addColumn(anotherArray);
>
> ...
>
> Header h = AsciiTableHDU.manufactureHeader(d);
>
> AsciiTableHDU newAscii = AsciiTableHDU(h,d);

Now that we're ready to add it to a FITS object and write it to output.

We could also have used a setColumn (or setRow or setElement) method to
modify existing data. E.g.,

> Fits f = new Fits("aFileName");
>
> BasicHDU\[\] hdus = f.read();
>
> AsciiTableHDU ath = hdus\[1\];
>
> ath.setColumn(3, new float\[ath.getNRows()\]);
>
> ath.rewrite(someArrayDataOutput);

will zero out the fourth column of the table. (Remember we use Java
indices!). We're not allowed to change the type of the column, so the
original table had to have a float\[\] array too.

Another way of doing this would be:

> float\[\] row = (float\[\]) ath.getColumn(3);
>
> for (int i=0; i\<row.length; i += 1) {
>
> row\[i\] = 0;
>
> }
>
> ath.rewrite(...)

The array we get back in getColumn is actually the kernel array used to
store that column's information. So if we modify it, we modify the
underlying data. This is not true if the column is a column of Strings.
Then you really do have to use setColumn -- so it's probably safer.

**ASCII table null values:**

ASCII tables allow the user to specify that a value is null whenever the
ASCII pattern in the file matches a specified null pattern. To check if
an element is null you can use the isNull call. Also, the getElement()
and getRow() methods will return a null for null values. However numeric
columns return a single primitive array, so they just have a 0 for
nulls. You need to use the isNull methods to check if you find a if you
get data using getColumn or getKernel.

To set a field to null, just use the setNull method. Make sure that
TNULL is defined for the appropriate column.

**Precision in ASCII tables:**

The ASCII tables use the ByteParser and ByteFormatter routines to
provide efficient conversions between binary and ASCII representations
of numbers. These conversions may differ in the lowest bit from those
provided by the standard system utilities -- which are much slower.

When writing real values, ASCII tables use only the length field in
determining what to write. They ignore the number of decimals specified
for real numbers. This will likely be changed in the future but should
normally mean more accurate representations and fewer occasions where
numbers overflow their fields. The tables are likely to be less neatly
aligned however.

When creating header information for ASCII tables, the headers use a
default value for the TFORM for each type. I10, I20, E16.0 and E24.0 for
integer, long, float and double numbers respectively. Strings are
checked for the longest string and are formatted for that to fit. Users
can override these values as desired by specifying the TFORM values
explicitly. If rows are added to ASCII tables the existing field lengths
are preserved. If String cannot fit into the space provided it is
truncated. If no sensible representation of a number can fit into the
space provided a set of asterisks is inserted.

**Short Integer Types in ASCII tables:**

There is currently no support for byte, short or char arrays in reading
or writing ASCII tables.

**Reading Binary Tables:**

Reading a binary table is very similar to an ASCII table, but the kernel
data type is very different. The kernel is a new class called a
ColumnTable. When I first attempted to read binary tables I found that
they were extremely slow. There were too distinct reasons: First, binary
tables often are comprised of small heterogeneous elements, e.g., an int
followed by a double\[2\] followed by a short. Thus the ArrayDataXput
methods cannot read the data efficiently. These are designed to be
efficient when the data is comprised of large arrays.

The second problem is that if we store the data naively, with each
element in the array as separate object, we may need to create millions
of objects. Just creating these objects can cause a noticeable pause in
a FITS-reading program.

We don't have this problem with ASCII tables because it's quite easy to
represent each column in the table as an array, so there are only as
many objects as there are columns. In ASCII tables each element must be
a scalar, but for binary tables each element can be an array of any
dimensionality.

The ColumnTable addresses these two issues. It stores data in a fashion
similar to the FITS AsciiTable class, each column is represented by a
single one-dimensional array. However each column also has associated
with it an integer array describing the dimensionality of the array. The
ColumnTable can read and write complete rows in a single function call.
The ColumnTable is not a FITS class. It can be used to read and write
other binary tabular data. When users actually retrieve data from a
ColumnTable they can choose to either get it as the one-d array or to
'curl' elements into their nominal dimensionality.

Let's take a look:

> Fits f = new Fits("binaryTableFile");
>
> BinaryTableHDU h = (BinaryTableHDU) f.getHDU(1);
>
> Object\[\] row23 = h.getRow(23);
>
> Object col3 = h.getColumn(3);
>
> Object col3f = h.getFlattenedColumn(3);
>
> Object elem = h.getElement(10,20);
>
> ColumnTable t = (ColumnTable) h.getKernel();

Note that the getRow method returned an array of objects, one for each
column in the table. If an element is itself an array this will be
appropriately curled.

The getColumn call returned a curled array. E.g., if the third column is
a 3x2 array and there are 500 rows in the table, the col3 might be
int\[500\]\[3\]\[2\]. Unlike getColumn in the AsciiTable methods, we
can't modify the FITS file by looking at this array. This is a copy of
the internal data, not a reference to it.

The getFlattenedColumn is a little more efficient. For the same example
it would return an int\[3000\] which is how the data is stored
internally. Indeed, the getFlattenedArray returns you a pointer to the
actual FITS array data. You can modify the FITS file by modifying this
array -- we're getting this array directly from the ColumnTable kernel.

The getElement returns a single element, curled if the element is a
multidimensional array. Even if the column contains scalars, the
returned object will be a length=1 array. The scalar types like Integer
are never returned by the FITS classes. However a scalar String's will
be returned as a String, not a String\[1\].

The getRow and getElement methods allow deferred reading, but the
getColumn methods read in the entire table. The getKernel method also
ensures that the all data are in memory.

**Reading a binary table by row:**

Both binary and ASCII HDU types extend the type TableHDU. Binary tables
support the same basic get, set and add operations as ASCII tables.
Users can also access binary table data at a lower level. Here's one
idiom.

> BufferedDataInputStream s = new BufferedDataInputStream(...);
>
> Fits f = new Fits(s);
>
> f.getHDU(0); // Read primary HDU.
>
> Header hdr = Header.readHeader(s); // Read table header
>
> BinaryTable bt = new BinaryTable(hdr);
>
> Object\[\] row = bt.getSampleRow();
>
> int nrow = hdr.getIntValue("NAXIS2");
>
> for (int i=0; i\< nrow; i += 1) {
>
> s.readArray(row);

... Process a line.

> }

Each element of row is an array of the appropriate type and
dimensionality for the binary table elements.

**Reading variable length columns:**

While not strictly part of standard FITS variable length columns are
extensively used by some organizations (mine included). These allow a
table to have a column that varies in length row by row. Variable length
columns are only supported as one-dimensional arrays.

Variable-length columns are handled automatically by the FITS reader.
Users may note that one or more elements might have zero-length. The
isVariableColumn method allows the user to see if a column is specified
as variable -- but does not actually check if it varies.

Note that the data for variable length columns is not stored in the
ColumnArray that is returned as the kernel of the binary table. There is
a FitsHeap object associated with any table with variable length
columns. The BinaryTableHDU add, get, and set methods handle variable
length arrays automatically. If the user wants to do devious tricks,
e.g., have many rows in a table point to the same binary table data,
then methods in ColumnTable and FitsHeap will need to be called
directly.

When variable length columns are read, no aliasing information is
retained. A distinct array is created for every row in the table when
the variable length column is read, even if the data was originally

aliased.

**Writing binary tables:**

A binary table HDU can be created from two distinct types of data arrays
of Objects and existing ColumnTables that the user has created. An
Object\[\]\[\] arrays is used to specify each element of the table.

An Object\[\] array is used to specify a binary table as an array of
columns in much the same fashion as ASCII tables. Note that some caution
is needed to ensure that the appropriate constructors are called. If a
binary table is to be composed of simple arrays, then the user may want
to disable the creation of ASCII tables -- or create the BinaryTable HDU
directly by calls to the appropriate functions and constructors in
BinaryTableHDU.

Users can also read or write a binary table line by line. E.g.,

> Object\[\] row = new Object{new float\[3\],
>
> new int\[3\],
>
> new short\[1\],
>
> new double\[500,500\]};
>
> Object\[\]\[\] table = new Object\[1\]\[\];
>
> table\[0\] = row;
>
> Header h = BinaryTableHDU.manufactureHeader(table);
>
> h.setNaxis(2, nrows);
>
> BufferedDataOutputStream s = new BufferedDataOutputStream(...);
>
> ImageHDU.getNullHDU().write(s);
>
> h.write(s);
>
> for (int i=0; i\<nrows; i += 1) {
>
> ... fill up with the next row of information...
>
> s.writeArray(row);
>
> }
>
> int pad = FitsUtil.padding(nrows\*ArrayFuncs.computeSize(row));
>
> s.write(new byte\[pad\]);
>
> s.flush();
>
> s.close();

First we create a sample of the row we want. We then create an
Object\[\]\[\] array with our row as the only row. Then we use this to
create a binary header. The call to setNaxis sets the number of rows in
the header.

Now we're ready to begin writing so we set up the output stream, spit
out a null HDU and the modified header finally write as many rows as we
want.

When we're done we need to make sure to add padding at the end of the
data.

**Efficiency and writing row by row**

The addRow methods of AsciiTableHDU and BinaryTableHDU allow the user to
dynamically add rows to an existing table. Users are warned that these
methods are typically very slow and are only intended for situations
where a few rows are being added to a table. The mechanism described
above can be used for writing binary tables row by row, but there is
currently no good way to construct an ASCII table in this mode. It this
case it's best for the user to collect all the information into columns
before writing the table.

#### Binary table internals

Users can treat ASCII and binary tables using much the same interface,
but internally they are very different. There are several different data
structures that are used internally within binary tables, and for the
most efficient access to binary tables a user needs to understand a bit
of this.

A binary table column is an n-dimensional array, where the first
dimension is the number of rows in the table. The additional dimensions
are what would be specified in the TDIM field for that column.

If a column is a two-dimensional primitive array the second dimension
may be variable. This is represented as a varying length column in the
binary table HDU. E.g.,

short byte\[\]\[\] = {{1},{1,2},{1,2,3}}

might be used in a varying length column with three rows.

A flattened column is a one-dimensional array where we have unrolled the
other dimensions. E.g., suppose we have TFORM8 = '80E' and
TDIM8='(2,2,2,10)' in a table with 100 rows. The column would be
represented as a float\[100\]\[10\]\[2\]\[2\]\[2\]. When converted to a
flattened column it would be a float\[8000\]. Flattened columns are used
to minimize the creation of array objects. Variable length columns
cannot

be flattened.

A model row is an exact image of how a row in the binary table is
stored. In the above example, it would consist of an Object\[\] where
the eighth element was a float\[10\]\[2\]\[2\]\[2\]. Note that model
rows represent how the data is stored in the FITS file. Since Strings
and booleans are converted into byte arrays on output to the binary
table, this is how they appear in a model row. Also, variable length
columns are represented not by their actual data, but by a two element
int descriptor array.

The object which is normally used to read and write binary table data is
the ColumnTable. This basically is a structure of flattened columns
transformed to the data type used in the binary table. I.e.,

String and boolean data has been converted into bytes, and variable
length columns have been converted into descriptor arrays.

Variable length data is stored in a FitsHeap. Whenever data is written
to a variable length column the heap is extended. Whenever a user reads
variable length columns, the descriptor information is used to extract
data from the heap.

**Unknown data types.**

The FITS library supports the reading and writing of data types that are
not fully understood by the FITS library. Any FITS extension that
follows the rules for indicating the size of the HDU can be read. Any
data structure that can be written using the writeArray method of
ArrayDataOutput can be encapsulated in a FITS extension. This includes
multi-dimensional (possibly non-rectangular) arrays of primitives,
Strings or Objects, where any Object must itself be one of these types.
Using Object arrays an arbitrarily complex structure can be supported.

Data is read into a byte array or is written using an
XTENSION='UNKNOWN'. Note that one cannot trivially save and restore data
in a FITS file. If a user has some structure it can be written to a FITS
file using the Unknown type, but to read it back the user will need to
be able to recreate the data structure without help from the FITS file.

Users may find these classes useful if there are non-standard extensions
in their FITS data. The FITS reader can successfully bypass these
unknown types and read later extensions. Also see the section on
trimming the installation.

**Re-writes.**

HDU's and their constituents can be re-written if the data is on a
local, non-compressed file. E.g.,

> Header h = someHDU.getHeader();
>
> int oldSize = h.getNumberOfCards();
>
> ... Modify header ...
>
> int newSize = h.getNumberOfCards();
>
> if (newSize \< oldSize) {
>
> for (int i=0; i\<newSize-oldSize; i += 1) {
>
> h.addCard(new HeaderCard("DUMMY", null, null);
>
> }
>
> } else if (newSize \> oldSize) {
>
> Cursor iter = h.iterator();
>
> while (iter.hasNext() && newSize \> oldSize) {
>
> if (iter.next.getKey().trim().equals("DUMMY")) {
>
> iter.remove();
>
> newSize -= 1;
>
> }
>
> }
>
> }
>
> if (newSize \> oldSize) {
>
> ... do something since rewrite won't work...
>
> } else {
>
> // Output modified header.
>
> h.rewrite();

}

The user might fill up a header with lots of DUMMY's to start with to
ensure there is space for new cards later. We're a little cautious
above. The number of header cards needn't be strictly constant , just
the expression (ncard+35)/36 (the number of 2880 byte blocks in the
header).

#### Objects and I/O Streams

In the examples above the user may be confused since we sometimes have
calls of the form

hdu.write(stream); and others

stream.write(data);

When do we send the object to the stream, and when do we send the stream
to the object? Basically if the object is a FITS object, it will take a
stream as an argument for input or output. If the object is a Java
array---comprised of only Objects, Strings, and primitive types, then
this object can be sent to the DataArrayXput implementors for reading or
writing.

## Changes between V0.9 and V0.91

**FitsDate:**

\- added getFitsDateString

**Header:**

\- made several methods public.

\- added checking for initial keywords before write.

**BinaryTable:**

\- removed TDIM keywords for variable length columns.

\- fixed bug that made BinaryTable(Object\[\]\[\]) constructor unusable.

**BinaryTableHDU:**

\- fixed usage of THEAP keyword

**AsciiTable:**

\- use blanks for data filler rather than nulls.

**BasicHDU**

\- made getDummyHDU public.

**HeaderCard**

\- fixed padding of string values which sometimes had one too many
spaces.

**image.ImageTiler**

\- allow requests for tiles that are not fully within the original
image.

**util.ByteFormatter**

\- changed formatter to use \'E\' (rather than \'e\') for exponents
since \'e\' not legal for FITS ASCII tables.

## Changes betweenVersion 0.6 and 0.9

There are many changes in between in this release. Changes to the public
interface are detailed below class by class.

**package nom.tam.util:**

> **BufferedDataInputStream:**
>
> The readPrimitiveArray call is deprecated in favor of the readArray
> call.
>
> A set of read(primitiveArray, start, offset) methods are now provided
> for the 8 primitive array types.
>
> The ArrayDataInput interface is implemented.
>
> **BufferedDataOutputStream:**
>
> The writePrimitiveArray call is deprecated in favor of writeArray.
>
> A set of write(primitiveArray, start, offset) methods are now provided
> for the eight primitive types.
>
> The ArrayDataOutput interface is implemented.

### ArrayDataInput/ArrayDataOutput

These are two new interfaces which indicate that the implementing class
can read and write arrays of data.

### RandomAccess

This new interface indicates that the implementing class can do random
access input.

### BufferedFile

This new class implements the ArrayDataInput, ArrayDataOutput and
RandomAccess interface. It provides a layer on top of RandomAccessFile
(though it does not extend it) to provide efficient reading and writing
of data with a random access capability.

### HashedList

This new class provides the functionality for traversing and
manipulating FITS headers.

**ByteFormatter/ByteParser**

These new classes provide ASCII-Binary conversions for standard data
types.

**ArrayFuncs**

ArrayFuncs now implements the PrimitiveInfo interface which provides a
single location for storing information about primitive types.

An nElements method which counts the number of elements in a (possibly
multidimensional) array was added.

The new getBaseArray method given a multidimensional array returns the
one-d array comprising the initial elements of the input. E.g., given
int\[5\]\[6\]\[7\] it will return an int\[7\].

The convertArray method which copies an array into another which may be
of a differing type, was broken into two pieces: mimicArray and
copyInto.

> mimicArray creates an array of the same dimensionality as the
> original, but with a possibly different type.
>
> copyInto, which does a kind of generic Arraycopy to handle multiple
> dimensions and differing primitive types.

The convertArray method is still available.

**PrimitiveInfo**

This new interface brings together some basic information on primitive
types and arrays.

**package nom.tam.image**:

This is a new package comprised of the single ImageTiler class. The
Image Tiler is used to provide efficient access to subsets of images.

**package nom.tam.fits**:

**Deleted classes**:

> **HDU** \-- The functionality of the HDU class is now divided among
> the FitsFactory, Fits and XxxHDU classes.
>
> **Column** -- Columns are no longer used. The primary utility of
> columns was to support things where a user would manipulate columns
> (move them around in a table). This functionality is no longer
> anticipated for the FITS library. The Header.positionAfterIndex method
> may be useful to ensure that column keywords are appropriately placed
> in the header.

**PrimaryHDU** -- This class was deleted. All primary HDU's are now
treated as ImageHDUs. It seemed a violation of object oriented
principles that the class of a given object would depend upon its
position in the FITS file rather than any internal characteristic.

**Modified classes**:

> **Many classes**:
>
> Except for constructors in the Fits class, all references to I/O
> streams are now through the
>
> ArrayDataInput and ArrayDataOutput interfaces. Since the
> BufferedDataXputStream classes that were previously used now implement
> these interfaces, no changes to existing code is required to
> accommodate this change.
>
> **Fits:**
>
> The FauxHDU class has been deleted (replaced by UndefinedHDU).
>
> The SkippedHDU class has been deleted. The intent here is that if a
> user wishes to skip an HDU it should not hang around. The deferred
> input capabilities mean that there is no reason to skip a class just
> for efficiency issues.
>
> There is no main method (all test methods are in nom.tam.fits.test
> classes).
>
> The method getInputStream has been added.
>
> The makeHDU(Header h) and makeHDU(Object o) methods have been added.
> (These
>
> replace functionality in the HDU class).

**BasicHDU**

> There is no explicit constructor for BasicHDU's (since BasicHDU is
> abstract this should not matter to users).
>
> The getFileOffset() method is added. It returns the beginning offset
> of the HDU (or --1) if the input is not random access.

The getKernel() method is provided to replace getData().getData() as a
convenience.

> A set of addValue convenience methods are available to add entries
> into the header rather than doing getHeader().addValue(...)

The rewrite and reread methods allow users to update or refresh HDU's
read from a RandomAccess stream.

### Data

> The getFileOffset method returns the offset of the data element within
> a RandomAccess stream or --1 otherwise.
>
> The reread and rewrite methods allow data to be revised or refreshed
> from/to RandomAccess streams.
>
> The getTrueSize and getPaddedSize methods have been deleted.
>
> The getData method has been deprecated in favor of getKernel. It is
> confusing that getData does not in fact return a Data object).

### ImageHDU

> The only constructor for ImageHDU's combines existing Header and Data
> elements.
>
> Static manufactureData, manufactureHeader and encapsulate methods are
> added. These are referred to from the FitsFactory class.

### ImageData

Image data now includes an inner class which extends the ImageTiler.
This Image tiler object can be retrieved using the new getTiler() method
and then used to extract subsets from the image. If the entire image has
not already been read in, the tiler will only read in the portion of the
image requested for each tile. If the entire image has already been read
in the tiler will generate a subset of that image.

### RandomGroupsHDU

Static manufactureData, manufactureHeader, and encapsulate methods were
added for use by the FitsFactory class.

### RandomGroupsData

> The RandomGroupsData(header) constructor was deleted. (It is redundant
> with RandomGroupsHDU.manufactureData().
>
> The getPadding methods was deleted (see FitsUtil.padding)

### BinaryTableHDU

> BinaryTableHDU now extends TableHDU.
>
> The getNumCols and getNumRows were replaced by getNCols and getNRows
> (in TableHDU (adopting a consistent name for this function among the
> FITS classes).
>
> All methods taking or returning Column objects were deleted.
>
> Static manufactureData, manufactureHeader and encapsulate methods
> where added for use by the FitsFactory class.
>
> A consistent set of get, set and add methods are provided by the
> TableHDU class.
>
> **BinaryTableData**
>
> Columns are now set, added or gotten using Object\[\] arrays rather
> than Object\[\]\[\] arrays. This allows simple primitive arrays to be
> used in a natural way.
>
> The getNrow and getNcol are replaced by getNCols and getNRows.

The getFlatColumns method returns an array of flattened data.

### Header

> Header has undergone very substantial reorganization. The KeyHash and
> KeyChain classes used previously have been replaced by use of the
> HashedList and HashedListIterator classes in nom.tam.util.
>
> addXxxValue methods have been replaced by addValue.
>
> the size() method has been deprecated since it is unclear whether size
> refers to the number of bytes or cards. The getNumberOfCards method is
> recommended.
>
> There is no Header constructor to generate a Header from a data
> element. Use FitsFactory or the manufactureHeader methods of the
> XxxHDU classes.
>
> Methods to get long integer values are now provided.
>
> The various methods relating to setting and finding the mark in the
> Header are deleted. The setKey method of the HashedListIterator
> replaces this (along with the add methods of HashedListIterator).
>
> The pointToData methods has been replaced by methods in XxxHDU .

**New Classes:**

### AsciiTableHDU/AsciiTable

\[I suppose these existed in the previous version but only as stubs\]

These provide support for ASCII tables.

**FitsHeap**

This class handles management of the Heap for BinaryTables.

### UndefinedHDU/UndefinedData

These allow the FITS class to read and write data to FITS files when the
underlying type or structure is now known.

> **FitsFactory:**

This class provides the mechanisms by which the system discovers the
types of FITS data, e.g., what kind of Header or data element is
present. It is also the location for any flags that need to be set to
control the operation of the FITS classes. Currently the only such flag
is useAsciiTables which controls whether an AsciiTable will be generated
from given data.

> **FitsUtil:**

Simple static functions useful in a number of locations in the FITS
library.

**Trimming the installation.**

The Java library supports all standard Java types. Unfortunately that
can make it rather cumbersome for users who only want to read simple
FITS images. To minimize the loading of classes that are not going to be
used a user can modify the FitsFactory class. The FITS factory
essentially defines the FITS types that will be available to the user by
default -- though users can explicitly use the other types if they wish.
Delete references in FitsFactory to the HDU formats that are not
desired. Unused classes will no longer be loaded. If only image data is
supported the total size of the Java library can be cut to about half
what is required for the full installation. This may be particularly
important if the FITS classes are to be used within an applet.