# Presentation: 099903

## Gfdnavi: A tool to archive, share, distribute, analyze, and visualize geophysical fluid data and knowledge

**Takeshi Horinouchi** (Kyoto Univ  Hokkaido Univ(soon)),
**Seiya Nishizawa** (Kyoto Univ  Kobe Univ),
- Chiemi Watanabe (Ochanomizu Univ),
- T. Koshiro, A. Tomobayashi, S. Otsuka,
- Y. Morikawa, Y.-Y. Hayashi, M. Shiotani, and
***GFD Dennou Club (Davis project)***

## What is Gfdnavi

- = Geophysical fluid data navigator
- A suite of software to construct Web-based database of geophysical fluid data
- Functionality:
- Search
- Data analysis and visualization
- Documentation of analysis results
- Available:
-     http://www.gfd-dennou.org/arch/davis/gfdnavi/

## Background

## Problems of Web-based database and analysis tools

- Limited analysis capability
-    We often end up with downloading data
- Not very suitable to desktop use
-    Service are not available to local data

## More on the analysis capability

- Impossible to predefine sufficient functionality  (since we are scientists)
-    Programmability is the key
- Programmability in two ways:
- Programmable on web-browser
- Web-service API (program locally)
-     Both are desirable

## Visualization is not the goal

- To others (scientists / society): reports
- While working: memos / internal documents
- To collaborators: reports / know-how / discussion

## Foundation of Gfdnavi

## Two fundamental libraries used to build Gfdnavi (open-source)

**GPhys** – a Ruby library to analyze and visualize geophysical fluid data *(by Horinouchi etc since 2003)*
- For consolidated access to data in files (NetCDF, GRIB, GrADS, NuSDAS, HDF5-EOS) or on runtime memory – A community infrastructure for data analysis    [http://ruby.gfd-dennou.org/] *(since 1999)*
**Ruby on Rails** – Development framework for Web application *(since 2005)*
- Made it drastically easy to develop Web applications with RDB
- Written in/for Ruby  We can use GPhys directly

## GPhys (Gridded Physical quantity)

- GPhys (Gridded Physical quantity)

**u = GPhys::IO.open(”u.nc”,”U”)**
**v = GPhys::IO.open(”v.ctl”,”V”)**
**uv = u * v **

-  in NetCDF [m/s]

-  in GrADS [m/s]

-  result on memory [m2s–2]

## Why do we use Ruby?

- Since we wanted a language for daily data analysis
- Easy (fast) to write
- Interactive use  like GrADS
- Python is also fine (but we love Ruby)

## Introducing Gfdnavi

- Early History (Aug 2006):
- Rough design by Horinouchi etc (at a meeting of the GFD-Dennou davis project)
- First implementation by S Nishizawa – In two weeks (since then he is the most contributing to its development)

*Since 2006*

## Overview

- User

- RDB (metadata etc)

- Browser Access

- Local file system
- (or opendap dir)

- meta data scan
- sync with DB

- access numerical
- data

- O/R mapping

- Web service

- Gfdnavi
- MVC
- core

- Web
- server
- (webrick/
- Apache)

## Metadata DB

- Directory tree

- group

- attributes

- data files

- variables (numeric data)

- supplementary
- text files

- description = “........”
- param1 = value1
- param2 = [val21,val22]

- Metadata
- name-value attributes; with a few standard field names
- geospatial- and time-coordinate info
- size, user info etc
- Directory structure (inherit metadata from parent directories)
- Generated by automatic scan (with a command)
- variables: reading attributes through GPhys
- directories: directory name and “Readme”-type texts

- 1

- n

- 1

- n

- spatial_and_
- temporal_attributes

- keyword_attributes

- 1

- n

- variables

- directories

- 1

- n

## User Interface

**Home** : Independent simple html  replaceable

## Functionality

- Browse directory
- tree (Finder)

- Search
- (Explorer)

- View docs
- (Knowledge)

- Write knowledge
- document

- Select numerical data

- Select

- Typical work flow to use Gfdnavi’s browser UI

- Browser UI Header

## Functionality

- Browse directory
- tree (Finder)

- Search
- (Explorer)

- View docs
- (Knowledge)

- Write knowledge
- document

- Select numerical data

- Select

- MS Explorer-like
- tree

## Functionality

- Browse directory
- tree (Finder)

- Search
- (Explorer)

- View docs
- (Knowledge)

- Write knowledge
- document

- Select numerical data

- Select

- Free text

- Attributes

- Search with Google Maps

- Results

## Functionality

- Browse directory
- tree (Finder)

- Search
- (Explorer)

- View docs
- (Knowledge)

- Write knowledge
- document

- Visualize / analyze
- (Analysis)

- repeat

- Select numerical data

- Select

- Ruby Script & Minimum Subset Data

- Save in the DB (login needed)

- Get the URL to redraw the img

## Functionality

- Browse directory
- tree (Finder)

- Search
- (Explorer)

- View docs
- (Knowledge)

- Write knowledge
- document

- Visualize / analyze
- (Analysis)

- repeat

- Select numerical data

- Select

## Functionality

- Browse directory
- tree (Finder)

- Search
- (Explorer)

- View docs
- (Knowledge)

- Write knowledge
- document

- Visualize / analyze
- (Analysis)

- repeat

- Select numerical data

- Select

## Functionality

- Browse directory
- tree (Finder)

- Search
- (Explorer)

- View docs
- (Knowledge)

- Write knowledge
- document

- Visualize / analyze
- (Analysis)

- repeat

- Select numerical data

- Select

## Functionality

- Browse directory
- tree (Finder)

- Search
- (Explorer)

- View docs
- (Knowledge)

- Write knowledge
- document

- Visualize / analyze
- (Analysis)

- repeat

- Select numerical data

- Select

## Functionality

- Browse directory
- tree (Finder)

- Search
- (Explorer)

- View docs
- (Knowledge)

- Write knowledge
- document

- Select numerical data

- Select

## Functionality

- Browse directory
- tree (Finder)

- Search
- (Explorer)

- View docs
- (Knowledge)

- Write knowledge
- document

- Select numerical data

- Select

## Functionality

- Browse directory
- tree (Finder)

- Search
- (Explorer)

- View docs
- (Knowledge)

- Write knowledge
- document

- Select numerical data

- Select

## Functionality

- Browse directory
- tree (Finder)

- Search
- (Explorer)

- View docs
- (Knowledge)

- Write knowledge
- document

- Select numerical data

- Select

## Functionality

- Browse directory
- tree (Finder)

- Search
- (Explorer)

- View docs
- (Knowledge)

- Write knowledge
- document

- Select numerical data

- Select

## Functionality

- Browse directory
- tree (Finder)

- Search
- (Explorer)

- View docs
- (Knowledge)

- Write knowledge
- document

- Select numerical data

- Select

## Functionality

- Browse directory
- tree (Finder)

- Search
- (Explorer)

- View docs
- (Knowledge)

- Write knowledge
- document

- Select numerical data

- Select

## Functionality

- Browse directory
- tree (Finder)

- Search
- (Explorer)

- View docs
- (Knowledge)

- Write knowledge
- document

- Select numerical data

- Select

## Functionality

- Browse directory
- tree (Finder)

- Search
- (Explorer)

- View docs
- (Knowledge)

- Write knowledge
- document

- Select numerical data

- Select

## Functionality

- Browse directory
- tree (Finder)

- Search
- (Explorer)

- View docs
- (Knowledge)

- Write knowledge
- document

- Select numerical data

- Select

## Functionality

- Browse directory
- tree (Finder)

- Search
- (Explorer)

- View docs
- (Knowledge)

- Write knowledge
- document

- Select numerical data

- Select

## Web service

-          Tomorrow by Seiya Nishizawa

## Network of Gfdnavi

- Under development by C Watanabe (Ochanomizu Univ)
- To create peer-to-peer network for cross search and cross use **among Gfdnavi servers**
- Then one can access **local data and remote data together**

- RDB

- RDB

- RDB

- RDB

- RDB

- dataA

- dataB

- dataC

## Summary

- Novel features of Gfdnavi
- Seamless coverage from desktop use to public data service (by having custom web server)
- Programmability (on browser & by web service)
- Documentation of analysis results (dynamically reproducible/extendible) ( memos / reports / PR / **Blog for scientific collaboration**)
- Good implementation
- Extendibility (by using GPhys)
- Swift development (by using RonR)

- Tomorrow by
- S Nishizawa

## Future Outlook

- Support Networking  **Create a Web of scientific data & knowledge**
- Increase analysis & visualization functionality (many needed)
- Improve remote API accesses (tomorrow’s topic)

## fin

## GPhys (Gridded Physical quantity)

- VArray (Virtual Array) – Abstracts Data Storage(Can be in file(s) or multi-D Array on memory; can also be a subset or aggregation of (an)other VArray(s))

## Example of GPhys’s associated coordinates

- (GPhys)

- “temperature” (4D VArray)

- (Grid)

- (Axis)

- “x” (1D VArray)

- (Axis)

- “y” (1D VArray)

- (Axis)

- “z” (1D VArray)

- (Axis)

- “t” (1D VArray)

- “coordinate variables”,
- but can be simple indices

- Supports “coordinates”
- in CF convention

- Supports transformed grids, scattered data points, etc etc

## What is Ruby on Railshttp://www.rubyonrails.org/

- Web development framework in Ruby
- With RDBMS (Mysql, Postgres, SQL Server, SQLite etc)
- Strong prototyping (e.g. Model-View-Controller (MVC) stucture)
- Comprehensive library (covering Ajax and Web service)
- Ruby-embedded html
-  suitable to use our Ruby library
- Has a private web server (Webrick); also runs on Apache, lighttpd etc
-  One can personally run a web server anywhere with arbitrary port

## From “Understanding Rails MVC”:http://wiki.rubyonrails.org/rails/pages/UnderstandingRailsMVC

## Sister-server method

- Register
- as sister

- User’s own
- Gfdnavi

- Use

- Indirect
- Use

-  Basic case: available in LAS.
-      User can’t choose peers

- (b) Gfdnavi: one can register
-     any peer by running a Gfdnavi

- Use

- Indirect
- Use

- Register
- as sister

- Register
- as sister

## P2P with directory server

- Direct
- Use

- Indirect Use

- Directory Server

- Query
- Server
- List

## Overlay network by P2P

- P2P Net

- Use

- Indirect Use

- Currently tested by C. Watanabe by using
- Overlay Weaver (Java-based p2p library)
- and Rails’ Action Web Service
-  – Decentralized p2p with distributed hash tables (DHT)

## copy from old slides

## GPhysA class of gridded physical quantities

- Takeshi Horinouchi (RISH, Kyoto Univ.)

- last revised: 2004/06/08

## VArray

- Virtual Array. A class of Ruby (written in pure ruby), which represents array data in GPhys
- A VArray object behaves as an array, but its contents can be on various media: (case 1) simply a multi dimensional array on memory (NArray),  or data in a NetCDF file (in this case, a file pointer is stored), or GrADS data; (case 2) It can also represent a subset of another VArray or multiple VArrays tiled.
- Can have attributes as variables in NetCDF datasets
- In reality, NetCDF are handled by a subclass VArrayNetCDF etc.etc.

- a VArray

- has  1

- array (NArray, NArrayMiss,
- NetCDFVar, or GrADSVar)

- has  1..* (multiple)

- VArray (whole or subset)

- a VArray

**Case 1**

**Case 2**

## subset mapping of VArray

- Always kept direct by compositing mappings, in order to prevent long chains (see the figure below).
- Subset slicing (by such as va[0..10,3]) is done by subset mapping, not by making actual data extraction, if not explicitly specified otherwise. Therefore,
- Computationally efficient
- Suitable for writing in subsets of data in files.
-  In other words, actual data cutting is deferred until needed – to defer operations until needed is a policy of GPhys construction

- VArray

- VArray

- VArray

- mapping  ***f***

- mapping ***g***

- mapping ***f *****。*****g***

## Structure of GPhys

- Consists of a grid (coordinates) and multi dimensional array data
- Can conduct mathematical operations (a GPhys behaves like an numeric array)

## For your reference: Coordinates in NetCDF dataset

- Variables that have same names as dimensions hold coordinate values (locations)
- Weak point: this rule can be violated

**var ****T**** **
**(4D; temperature)**

**var ****x**** **
**(1D; lon)**

**var ****y**** **
**(1D; lat)**

**var ****z**** **
**(1D; altitude)**

**変数 ****t**** **
**(1D; time)**

**dim ****x **
**(len=128)**

**dim ****y**** **
**(len=64)**

**dim ****z **
**(len=50)**

**dim ****t**
**(unlimited)**

**var ****Ps **
**(3D; **
**surface prs)**

**sample**

**Can construct GPhys objects along the trees**

## More on cooridnates

- 3 cases are prepared
- point sampling
- cell type
- simple sequence (though it’s not physical)

- For instance, how to integrate along an axis is known by the axis. GPhys simply requests the integration to its Grid, and the Grid ask it to the corresponding Axis. By default, trapezoidal formula is used if point samples or cell boundaries. (can be changed by users)
- If NetCDF data are read, those types are configured if the convention used supports such discrimination. (so far, convention support is weak, though)

## Tiling

- Data divided into “tiles” can be treated as one consolidated GPhys object.  Convenient to handle long time sequence divided by periods (such as by years) or outputs from parallel simulations on distributed-memory machines. Tiling is done by VArrayComposite.
- Subsets can be handled (see the figure below)
- May be applicable to parallel simulations in future?
- So far, automatic configuration is available only for NetCDF, by using an Array or Regexp (e.g., /data_x(\d)y(\d).nc/ for data_x0y0.nc, data_x0y1.nc, data_x1y0.nc, data_x1y1.nc)

- Subset specification
- of tiled data

## Big data handling

- Iterator to handle data too big to read on memory at once.
- GPhys::IO.each_along_dims_write – the result also written in file (since the result of operations is often big too.)  Another type of iterator is planned but yet to be implemented.
- Example:
- Without the iterator:
-   **in = GPhys::IO.open(infile, varname)**
**    ****ofile = NetCDF.create(ofilename)**
**    ****out = in.mean(0)          ****#  now, the entire result is on memory**
**    ****GPhys::IO.write( ofile, out )**
**    ****ofile.close**
- With the iterator, taking the last dimension to make a loop:
**  ****in = GPhys::IO.open(infile, varname)**
**    ****ofile = NetCDF.create(ofilename)**
**    ****out = GPhys::IO.each_along_dims_write(in, ofile, -1){ |in_sub|**
**                ****[ in_sub.mean(0) ]   ****#  written in *****ofile***** each time**
**              ****}**
**    ****ofile.close**

## Units of physical quantities

- Handled by NumRu::Units (by E Toyoda)
- mlt,div,etc.: handled as should be
- add,sub: the units of the first term is inherited
- e.g., addition of [m] and [km] is done after multiplying the second term by 1000.  Warning is made if the units are incompatible (in that case, no conversion is made).
- Introduced a scalar numeric class with units UNumeric
- GPhys, VArray, and UNumeric recognize one another (stronger to weaker in this order)
- Example: to multiply the Coriolis parameter with a GPhys object u representing winds [m/s]:
-     f = UNumeric[1e-4,”s-1”]
-     coriolis_frc = f * u      # then the units will be in m.s-2

## Distributed objects using dRuby

- Data service to remote clients
- gphys-remote: a simple directory service (like the anonymous ftp, directories and data (in which GPhys objects can be defined) under a top directory is made accessible to remote hosts.
- gave (GUI): can connect to gphys-remote server