NOTES FROM THE EIGHTH FORMAL COORDINATED ENERGY AND WATER-CYCLEOBSERVATIONS PROJECT (CEOP) TELECONFERENCE SATELLITE DATA ISSUES HELD ON
16 MARCH 2010
FINAL DRAFT, 15 July 2010
1.

INTRODUCTION

The Eighth CEOP Satellite Data Teleconference took place on Tuesday 16 March 2010 at 13:00
UTC. The issues that were brought up and discussed on the subject conference call included:
1.
2.
3.
4.

WCRP/GEWEX News and Issues
Distributed Data Archive
Satellite data formats and metadata
Satellite data providers reports – perspectives with respect to CEOPParticipants
The participants were:Toshio KoikeDennis LettenmaierPeter van Oevelen
Katsunori Tamagawa
Gang Ye
Michael Theobald
Sam Benedict
Petra Koudelova

Japan, CEOP Co-Chair & Satellite Data WG Chair
USA, CEOP Co-Chair
USA, International GEWEX Project Office
Japan, Asia region data management
USA, NASA MODIS Team
USA, NASA AIRS Team
USA; CEOP International Coordination Function
Japan; CEOP International Coordination Function

Drs Christopher Lynnes (Representing NASA AIRS team), John Bates (Representing NOAA NESDIS),
Satoko Miura (Representing JAXA), Yoshiyuki Kudo (JAXA/RESTEC); Kazuo Umezawa (RepresentingJAXA), Einar-Arne Herland (Representing ESA Earth Science Division), Michael Teague (Representing,NASA MODIS Team), Steve Williams (Representing CEOP Data Management), and Ed Kearns(Representing NOAA NESDIS) were not available for this call.

2.

NEXT CONFERENCE CALL

The next, 9th CEOP International Satellite Data Teleconference will be held on Thursday 15
July 2010. Benedict and Koudelova have the action (A1) to inform the group of the details of the next callnearer to the time of the call and to coordinate the origination of the call (action A1a). In this context, acalendar of the dates and times of all the CEOP conference calls for 2010-2011 was distributed to theparticipants for their future reference.
3.

SATELLITE DATA GROUP GENERAL ISSUES

3.1 WCRP and GEWEX related issues(3.1a) Lettenmaier, Koike, and van Oevelen reported on the GEWEX SSG meeting in New Delhi,
India in January and advised the group that Dr. Kevin Trenberth had been appointed new Chair of theSSG. They voiced that the SSG very well acknowledged the CEOP data component, including model output
as well as reference site and satellite observations. The SSG members recognized the value that the longterm high quality CEOP data products has for the science and thus continuation of this CEOP activity wasconsidered as highly desirable. At the same time, it was felt that the value of the CEOP integrated datasetsneed to be better advertised through accomplished studies using these data.
rdLettenmaier and Koike reiterated that at the 3 Annual CEOP Meeting in Melbourne in August 2009,CEOP took commitment of developing 10-year dataset that is especially needed for climate projection
studies focusing on climate model uncertainties as well as for intensive integrated studies in regions thatinvolve RHPs and isotope and modeling groups. The CEOP 10-year dataset should include in-situ as well
as satellite data and added will be data from other projects like the FLUXNET and IGBP iLEAPS data.

1(3.1b) In this context, Lettenmaier brought up an idea of a workshop focused on possibilities of utilizationof satellite data for model improvement.
nd(3.1c) Van Oevelen informed the group that the 2 Pan-GEWEX meeting that will take place in Seattle,
USA, 23 – 27 August 2010 (http://www.gewex.org/2010pangewex/home.html). The Pan-GEWEX meeting
will address how the GEWEX panels and their projects and working groups will continue to work over thenext 2 years to achieve their short-term goals, and how they will evolve to accomplish post 2013
Imperatives. This process will include determining what enabling infrastructure is necessary and developinga strategy for dealing with the GEWEX and WCRP cross cutting or overarching themes.
According to the updated Pan-GEWEX agenda, two full-day sessions on Tuesday 24th and Wednesday 25thAugust are designated for each panel (including CEOP, GRP, and GMPP) and one CEOP evening session
on Thursday 26 August has been added on the CEOP request. In addition, one day for panel interaction isscheduled on Thursday 26 August. Further information including logistics details can be found at themeeting website. The participants on the call were encouraged to consider their participation in this event.
(3.1d) It was reiterated that new mailing lists were created using the NCAR Mailman service with intention
to facilitate the CEOP groups communication and material distribution. The address for the Satellite datagroup is: ceop-satellite@eol.ucar.edu. To add/remove a person to/from the list, request should be made to
Steve Williams or Petra Koudelova and Sam Benedict.

3.2 Data format and metadata issues(3.2a) Koike advised the group that the JAXA team had developed a new metadata design because acertain problems had been found in the previous metadata format. The team has been working onconverting the old version metadata to the new one for all the JAXA and NASA MODIS as well as AIRS
data already available in the Tokyo archive. The new metadata format software will be provided to the AIRS
team to generate the correct metadata files for the data to be submitted to CEOP in future.
(3.2b) In this context, Tamagawa reported that a document metadata registration system had beendeveloped and was available as part of the Japanese DIAS services that enables an easy on-line input of
metadata using a user-friendly interface. Subsequently Tamagawa distributed the metadata format being
developed under the DIAS scheme.
The input items/information of meta-data registration system which was developed under the framework of
Data Integration and Analysis System (DIAS) consists of six mandatory items and six other items as notedin Appendix A.
(3.2c) Koike further voiced that JMA and JAXA would convert/provide their satellite products to theNetCDF format in a next few years. Ye mentioned that since the format conversion into NetCDF was not atrivial task, the NASA MODIS team planned to stay with the HT-4 format and update to the HTF-5 format infuture.
3.3 JAXA CEOS/WGISS Test Facilities (WTF) for CEOP
(3.3a) Koike advised the group on behalf of the JAXA WTF for CEOP team that a budget has beenallocated for expansion of their Distributed Data Integration system’s capabilities and inclusion of the CEOP
Phase 2 data. The task should be completed by the end of 2010.
To accomplish this upgrade of the system, the team plans firstly to develop a suitable design of the newfeatures. For that purpose, a special teleconference was held in February that included representatives ofthe WTF-CEOP JAXA team and individual data centers, namely MPI/DKRZ, NCAR/EOL, and UT. The
technical aspects of the data archives organization and connection between the WTF-CEOP JAXA systemand the centers were discussed and resolved to the extent required by the JAXA team for initiating further
development of their system. The development work was progressing as planned and possible issues werebeing communicated with individual data centers.

2

4.

THE AGENCY REPORTS

4.1 JAXA report
Koike reported on behalf of Umezawa that JAXA team was now focusing on reproducing the metadata files
as mentioned above in 3.2a. They also continue processing of the NASA MODIS data and have begun towork on the ESA data, which also meant remarkable increase of workload for the team. Therefore the JAXA
team would much appreciate if the MODIS team could accept the responsibility for processing their datausing the JAXA software and provide JAXA already processed products. The task of subsetting theproducts for the reference sites and monsoon regions would be done by JAXA.
In response, Ye mentioned that the MODIS team would consider this possibility but would need to do an
estimation of required effort and resources after they review the JAXA software. The JAXA (Umezawa) andMODIS (Ye, Teague) have action A2 to communicate this issue.
4.2 NASA MODISYe reported that the preparation of the Terra MODIS data for the EOP-3 and EOP-4 had been completed
and the transfer to the JAXA site should be finished. When the successful accomplishment of this task isconfirmed by JAXA, the MODIS team will remove the Terra. The Aqua MODIS data for the EOP-3 andEOP-4 have been prepared and wait for transfer to the JAXA site. After that, the data for the subsequent
years will follow.
Subsequently, the Terra EOP-4 data downloading was confirmed to have been completed and the Aqua
EOP-4 data downloading was also confirmed to nearing completion. The next step was determined to be tostart the processing of Terra and Aqua EOP-3 data in the same way as EOP-4. Since users' priority of
Terra data is higher than the Aqua, Terra data processing should be started first.
A question was raised about the availability of the netCDF conversion tool which needed to be resolved by
the Tokyo Team (Action A3) otherwise the data would continue to be provided in HDF format.
The NASA Team confirmed that they could start EOP-3 Terra data processing in early July. As mentioned
earlier the EOP-3 Aqua data processing can not start before the Tokyo Team completes EOP-4 Aqua data
download/ingest, since the EOP-4 Aqua data must be removed from the ftp server first before staging any
new Aqua data.
In the meantime the issue of the long process that was necessary to reconcile the Aqua EOP-4 data wasraised. Since it took so long to receive the huge list of missing data after the Aqua EOP-4 data process was“completed” (total number of dataday of missing data was 158245!), a great deal of effort was expended tocheck, verify, and reprocess the missing data.
From the experience of the EOP-4 data processing, the NASA Team wished to consider a different
approach to processing of EOP-3 data. It may be that the Tokyo Team may try to order EOP-3 data fromthe NASA MODIS L1 and Atmospheres Archive and Distribution System (LAADS). As an example, the firstmonth of data for site R001 could be ordered from the LAADS website. When the ordered data is processed(normally a few minutes to few days, depending on if the ordered data is online or not), an email will be sent
advising from where the data can be downloaded. After completing the data download, it can be checked
for missing files. If everything is acceptable, the second month of data can be ordered from the LAADS site.
Although it was understood that it can take just a few minutes to order the data from LAADS website theTokyo Team noted that this process had been tried earlier and determined to be very intensive if data are
missing and also to set up requests for subsetted data.
An action (A4) for the next call was to set up a test case for using LAADS to determine the efficiency of theprocess versus the previous downloading technique that had been applied to the EOP-4 data.
4.3 NASA AIRSTheobald reported that the AIRS team was finishing the processing of the EOP-3 and EOP-4 data. Koike
asked if it would be possible for the AIRS team to give a matter priority to processing of the 2008 data
(February to July, in particular) because these are much needed for research purposes over the Tibetan
Plateau. Theobald noted that it was very good to hear that the AIRS subsets were being used successfully.
However, Theobald voiced that they would discuss this request and would inform the group accordingly.

3

Subsequently NASA management was contacted about the possibility of the AIRS Team doing additional
processing, and of the possibility of having to make changes to the associated metadata. It was requested
then that in order to make a decision, it was necessary to gauge the resources this would require, andwould like to ask these questions:
1. Are both monsoonal region and reference site subsets needed, and for the same set of AIRS
parameters?
2. We would like to avoid having to commit resources to update the metadata generation software. Would it
be possible for our partners to continue to convert our metadata upon receipt?
3. What observation dates are needed? I heard Feb 2008 - July 2008, but is that definitive, or just aprioritization of a larger observing period?
Action A5 is on the Tokyo Office to respond to these questions by the time of the next call.

4

APPENDIX A:
The input items/information of meta-data registration system which developed under the framework of Data
Integration and Analysis System (DIAS), consists of six mandatory items and six other items.
The mandatory items are as follows;
1. Title- Name- Edition- Abbrevation
2. Contact- Contact Person name- Contact Person Organization- Address- Tel, Fax, E-mail3. Document Authors- Name- Organization- E-mail4. Dataset Creators- Name- Organization- E-mail5. Date of Dataset- Event- Date6. Dataset Overview- Topic Category- Abstract- Temporal Extent- Geographic Bounding Box- Grid- Geographic Description- Dataset Keywords- Online Resource
The other items are ;7. Data Environmental Information
8. Distribution Information
9. Data Processing
10. Data Remarks
11. Use Constraints
12. References
The document metadata registration system can support to input these kind of information very easily andcan create a matadata document (dataset documentation) with common format and based on ISO standard.

5