Items brought up at May 2002 HUG Meeting which the HEASARC should address
 (compiled by Steve Drake 05/08/2002: please send updates/corrections to him)
    T= Them (HUG) comment or question
    U= HEASARC reply or action
Notice: I have not listed interoperability/compatability concerns here
        that really fall under the purview of the HEADCC rather than the HUG
  (1) T: We should investigate off-site storage of copy of HEASARC Archive.
      U: Nick should address this.
     Phil comments:
  "Are we talking about offline or online storage offsite?  There are a number
of things that could be done, but there is no money to do them.  Does the
HUG consider the NSSDC MOU insufficient??  I'm not sure of the scope of
this proposed investigation."
     Steve D. comments:
  "There was a concern that if someone nukes GSFC, there goes the HEASARC
Archive and all its copies. I checked with Beth Brown of NSSDC and they
keep the tapes and CD's that we send them onsite also. Maybe the cheapest
thing to do is to see whether the NSSDC material can be moved offsite
(Laura Whitlock's basement?)."
  (2) T: Why the discrepancy in stats for Imagine the Universe site?
      U: Steve's number of >4 million is based on number of requested
         files, Jim's number of 0.5 millions is likely based on # of pages;
         remember that to download a page, a number of files are typically
         needed (apparently an average of 8 per page for this site). Steve
         will discuss this with Steve Fantasia to confirm.
  (3) T: Plot applet for Browse sometimes fails and/or hangs web browser.
      U: Tom and Christina should check this.
  (4) T: The search radius for cross-correlations cannot be adjusted or
         defaults to previous value.
      U: A bit obscure; Tom and Christina should check this or try and
         get clarification.
  (5) T: Exposure time is a very important parameter, and should be a
         default display parameter in all database tables in which it
         occurs.
      U: Ed should make it so.
  (6) T: In the ascii table option for results in Browse, the user should
         have the option of selecting the delimiter character (| should
         be default), with a blank being a possible value.
      U: Tom & Christina should make it so.
 (6.1)T: It would be nice to have an option for results in Browse to be
         put into a spreadsheet, e.g., an Excel table.
      U: We agree
         Tom comments:
  "During the meeting we mostly talked about how we might
play with the text format to make it easier to import
into Excel.  Christina has just spent a couple of days
looking at a more direct approach, providing a direct Excel-format
output from Browse.
The development version of Browse at
    http://olegacy/cgi-bin/w3bdev/W3Browse/w3browse.pl
now has an Excel compatible format as one of its
output format selections.  If you run Browse from a PC or Mac
you can pop up an Excel window directly from the browser.
Otherwise one can save the file and copy it to another machine.
This could be quite a useful capability and we'd be interested
in comments about the general idea and the implementation.
There are a couple of glitches that need to be worked out.
Most notably, we have a blank top worksheet -- each table
retrieved is placed in it's own worksheet but in general
things seem to be working quite nicely."
    Mike C.'s comments:
"it looks great and I think this is extremely useful already.  One
thing you might try is to embed links in the spreadsheet - for
example the column headers could be linked to the help files; rosat
sequence id's could be linked to the data - could you include links
to the services too  - that would be really neat
one minor thing - the dates seem to be given in (I'm guessing) MJD,
for example public_date = 50841.  That's fine (makes it nice and
sortable and calculable) but maybe this should be documented?"
    Tom's response:
"Putting the links for the help is easy enough.
Christina also thought we might use the top sheet
which is currently empty, to give help, which might
include a note on the format of dates and position
fields."
    Nick's feedback:
  "- people will not notice it, because it is buried in the output
selection menu, maybe these should be check boxes
   - the first spreadsheet should have the details of the query on it...
   - one thing that was asked for was a means to download the entire
catalog into excel, maybe we should provide that as an addition to
the tdat files (i.e. put them already made in the same ftp area, and
have a link to that from the browse page or the browse results page)."
     Tom's reply:
  "I agree it could be hard to find but checkboxes will use a lot of real
estate...  This would be the fifth distinct display format (HTML table, Text
table, XML, FITS, Excel).   I'm hoping that with this format we can avoid
putting in more options like 'text separator' and such. These aren't hard
to do, but just make the interface more and more complex.
One random thought I've had that might help this is provide a little tip at
the beginning of each Browse session.  E.g., randomly select
   "To query an entire table ..."
   "To get data in Excel ..."
   "You can get a GIF of a plot by ..."
   "To cross correlate two complete tables ..."
   "To list all tables by mission use ..."
   "The Browse Batch interface ..."
...
to advertise a number of the less well known features of Browse.  The frequent
users of Browse would then see all of these.  We'd have all of the tips on
some easy to read page so that a user who remembered there was a tip about
reading Excel data but not what it was could refresh their memory.
>
> - the first spreadsheet should have the details of the query on it...
... plus some explanations of the formats of the data and warnings
about Excel limits (256 columns, 65536 rows).
>
> - one thing that was asked for was a means to download the entire
> catalog into excel, maybe we should provide that as an addition to the
> tdat files (i.e. put them already made in the same ftp area, and have
> a link to that from the browse page or the browse results page)
I'd been thinking of putting the capability to download an entire table
on the Table Index page -- allowing any format.  It would be easy to do there,
but that's a fairly obscure page -- not that the TDAT files are better known.
It should be fairly easy to write a TDAT to Excel converter so that we could
add that to the process that updates the TDAT files.   Probably easy enough
to do both.
There are a fair number of tables that exceed the 64K row limit.  My initial
thought is just to give the first 64K rows for them.  The alternative is to
split the table into multiple spreadsheets."
  (7) T: Some mission database tables, e.g., XMM-Newton tables, have no
         class parameter field so that, for example, to find out how many
         AGN or CVs that have been observed is not simple, but requires
         finding a table of AGN or CVs, and doing a cross-correlation.
      U: This is a function of what info the mission has provided in their
         tables: XMM-Newton did not provide us classes (even though they
         did, I believe, ask proposers for them in their proposal submission
         process) in their tables, and that's why they aren't there.
  (8) T: It would be very useful if there were an explicit high-level option
         to "Download All Rows of Table" in Browse (it is possible right now
         but requires going down to the Parameter Search level).
      U: This seems like a reasonable thing to do, but (i) should maybe be
         restricted to HEASARC tables only, and (ii) caveats should be given
         that this might result in a large download: it would be useful to
         explicitly list how many rows there are in the table, for example,
         so that the user would know how much (s)he was going to get back.
  (9) T: It would be good to expand and better document the HEASARC's nh
         webtool, e.g., give the original reference, list caveats such as
         the relatively low spatial resolution, the possibility of
         underestimates of total H column density in regions towards the
         Galactic Plane and molecular clouds where a significant fraction
         of the H may be molecular rather than atomic, adding other sources
         for nh such as the Elvis et al. (1994, ApJS, 95, 413) NEP survey,
         the various Lockman Hole surveys, the Elvis et al. (1989, AJ, 97,
         777) and Murphy et al.(1196, ApJS, 105, 369) values towards 173
         and 220 AGN and quasars, respectively, the Stark et al. (1992, ApJS,
         79, 77) Bell Labs Survey, the Hartman WSRT Survey, etc.
      U: The reference is actually given on the main Nh page and on the nh
         Help page; the latter discusses the 1 x 1 degree gridding used.
         Caveats can certainly be given more prominence. Adding other diverse
         sources for nh values is more problematical: it would be hard to
         combine pencil-beam surveys such as the AGN ones with all-sky or
         wide-angle ones. Possibilities include:
         (i) Making a Browse table or tables of Elvis et al & Murphy et al.
         and have a link on the nh page which would spawn a pre-formatted
         query to these tables.
         (ii) We could fairly quickly add the Hartman data to our tool
         [Steve Snowden has a copy of this dataset and the IRAS 100 micron
         data formatted in the same grid that he had the soft X-ray background
         data], and probably the Stark data.
         (iii) It's not clear to me what the best way to add the regional
         data such as the NEP and LH surveys would be.
 (10) T: It would be very useful to have an option when using the soft
         X-ray background tool to generate an output data of the model
         say for xspec.
      U: Steve Snowden says that he is writing an ftool to do something
         like this that should be ready in another few months. Maybe we
         could do this on the Web tool by having an option which would
         send a preformatted call to WebSpec?
 (11) U: Keith Arnaud presented an open question about whither XSPEC should
         go after version 12 is released
 (12) U: Steve Drake issued a call for input on the new HEASARC CookBook
         material on the Web. The HUG suggested that the X-Ray Summer
         School might be a good venue to test it, and Ilana Harrus has
         agreed to do this. The HEASARC booth at the Summer AAS Meeting
         might also be a good place and time. Rita Sambruna will have
         students working with her this summer who would also be good
         guinea pigs.
 (13) T: The Integral simulator is difficult to install.
      U: Chris should follow up on this.
         Chris comments:
   "I have little control over "OSIM", which is a collaborative
effort between ISDC and the INstrument teams. I did serve as a beta
tester, and gave them lots of feedback prior to its release (Ken did
 also).  I believe they are aware of the problems, but with the early AO
release, they simply could not fix it in time; it was of litte use.
What I can do here, certainly in advance of the next AO is implement
XSPEC/WebSPEC support for SPI (for which we are developing resonse and
background models), and hopefully implement IBIS in some more
approximate way as well. This does'nt address imaging though."
 (14) T: Are Integral calibration data publically available, and, if not,
         when will they?
      U: Chris should follow up on this.
         Chris comments:
  "Its a matter of project policy - I could easily put a large number of
SPI calibration spectra, in PHA or other simple format, on my FTP site
now. As far as the official ISDC archive, its a policy decision as to
when it becomes public."
 (15) T: There were several questions about the large amount of RXTE data
         downloaded: (i) How many people download RXTE data, is it a lot
         or is it dominated by a small number of expert users? (ii) What
         type of RXTE files do users download? Are they unnecessarily
         downloading the kitchen sink, when all they really need is a
         much smaller subset of files?
      U: Steve will follow up on this in consultation with Alan. Steve
         will (again) try to organize a meeting this month to discuss
         these and other RXTE Archive and table issues
 (16) T: A general comment was made about improving the notification of
         bugs or other problems to users of Xspec (presumably this could
         apply to other HEAsoft software), e.g., by having an option that,
         if enabled, when a user started up am Xspec session would query our
         site for bug reports released since the last time this user started
         up, and relay them to the user. Specific comments/requests were made
         concerning APEC models such as how up-to-date are they, or can
         they be tuned by the user, e.g., to let him/her add missing lines
         or change specific atomic data parameters?
      U: Keith is considering the bug notification idea. He noted that the
         APEC models used in Xspec are created by a code that is publically
         available, although it might not be trivial for the typical user
         to make changes and/or additions to its input data.
 (17) T: There is an urgent need to get the HETE-2 data in documented FITS
         formats which are compatible with existing software tools.
      U: We agree. Mike Corcoran has been actively trying to get the binary
         file formats from MIT so he can create FITS file versions of them,
         and will keep trying.
         On May 16th Mike sent this comment:
"Here is an update on the HETE2 archive.
I've been trying to figure out the MIT processing
pipeline and it's pretty slow going but I'm making a bit of progress.
I've copied over the relevant solaris binaries & scripts to
/FTP/hete2/ops, and (running on circe) can now split out the
instrument data from the archived telemetry files (in
/FTP/hete2/data/Archive).  That's the good news.  The bad news is
that the split files are still in ipp format.  I've identified
scripts which (apparently) create postscript lightcurves from the
split gamma-ray data but haven't yet figured out how to run them -
the problem is in trying to (partially) reproduce the MIT processing
environment, setting paths, etc, and most of the scripts are designed
to deal with the daily telemetry data, not the data in the archive
(which requires some modification to the paths, which need to be
modified anyway to run here and not at MIT!).  The idea (if MIT is
comfortable with it) is to split out the instrument data from the
telemetry files, then convert the split ipp data to either ascii
(there are scripts to produce qdp lightcurves for example) then to
FITS.  I'm not sure how long this will take though (if I can get some
response from MIT it will go quicker of course)."
 (18) T: It would be nice if the HEASARC could get a copy of the AGILE data.
      U: Nick discussed how the proposal to the 2000 Senior Review to do this
         was rated so low as to sink it. Lorella said that Agile's launch
         date is unlikely to be before 2004, in any event. Any further
         comments/suggestions?
 (19) T: Could there be fhelp-like help available for Xanadu tasks?
      U: Bill and/or Keith should address.
 (20) T: Could there be better help documentation for XSTAR?
      U: Steve relayed this to Tim Kallman.
 (21) T: There were concerns about the impact of NVO-related activities
         on the HEASARC budget, and whether they would adversely affect
         core HEASARC functions.
      U: Nick agreed that this was a worry, and that he was aware of this
         as a potential problem.
 (22) T: Could there be a new functionality in Browse so that one could
         request a specific data product such as a background map for,
         say, 500 observations, in a one-click shopping mode?
         [See attached note from Brandt amplifying this].
      U: Tom and Christina should address.
------------------------------------------------------------------------
Hi,
I wanted to write down the one final BROWSE comment that I only was
able to mention briefly at the end of the HUG meeting yesterday. I
can give this best as an example...
  Recently, we wanted to obtain the WGACAT FITS image files for several
  hundred ROSAT PSPC observations as cataloged by WGACAT. However, we
  could not find a "natural" way to do this in BROWSE, largely because the
  data are organized on a source-by-source or pointing-by-pointing basis.
  The only way to do it now seems to be to go through pointing-by-pointing,
  copying the one relevant file each time for each pointing.
  It would be helpful to have a "natural" and general way in BROWSE to
  access and copy the same _type_ of file for _many different_ data sets,
  so that problems like the above involving many different data sets have
  an easy solution.
I wrote down my other BROWSE comments yesterday and gave them to you;
feel free to contact me if there are any questions.
Thank you, Niel
---------------------------------------------------------------------------