# Presentation: 534903

## CUAHSI and CLEANER Overview:Infrastructure and Informatics

- Jon Duncan
- Program Manager

## Collaborative Large Scale Engineering Analysis Network for Environmental Research (CLEANER)

- Project Office funded, based in Arlington, VA
- Science, Cyberinfrastructure, Social Science, Sensors, Organization committees
- Major infrastructure planning process and more

## Consortium of Universities for the Advancement of Hydrologic Science, Inc.  (CUAHSI)

- A consortium of 105 research universities, 6 affiliate members, and 8 int’l affiliates
- Incorporated June, 2001 as a non-profit corporation in Washington, DC

## What is Hydrologic Science?

- Expands beyond traditional hydrology
- Focus on “why” the earth works as it does, like other earth sciences, moving beyond traditional problem-solving orientation
- Embraces parts of hydrology, geomorphology, hydrogeology, biogeochemistry, ...
- Hydrologic cycle is central organizing principle

## Purpose

- Science Objective: To further predictive understanding of the terrestrial hydrologic cycle and its linkages with climate and biogeochemical cycles
- Societal Need: Will there be enough water for the next century?
- ...of appropriate quality
- ...to meet society’s needs
- ...to maintain the integrity of our ecosystems

## Need for CUAHSI

- Larger-scale, longer-term research to advance science
- Enable research at disciplinary boundaries
- Support of larger research teams
- Improve efficiency and effectiveness of data collection and dissemination

## What CUAHSI is doing?

- Hydrologic Information Systems
- Observatories
- Hydrologic Measurement Facilities
- Education and Outreach
- Hydrologic Synthesis Center

## Abstractions in Modeling

- -Solution Techniques

- Physical World

- Conceptual
**Frameworks**

- Data
- Representation

- Model
**Representations**

- “Digital Observatory”

- Real World

*Measurements*

*Theory/Process Knowledge*
*Perceptions of this place*
*Intuition*

- Water quantity and quality

- Meteorology

- Remote sensing

- Geographically
- Referenced

- Mapping

- Validation

- DNA Sequences

- Vegetation Survey

- Hydrologist

- Q, Gradient, Roughness?

- Groundwater
- Contribution?

- Snowmelt
- Processes?

- Wetted Perimeter

## Data Representation

- Four-dimensional {x,y,z,t}
- Continental scope
- Multi-scale, multi-resolution
- Points, coverages, dynamic fields

## Digital Continent

- Integrating monitoring and research data yields a  single body of information for the country
- Observatories contribute intensive information to this body
- Observatories are placed within context of climate, geology, soils, etc. but are not assumed to be representative of an area.
- Digital observatories may be watersheds, aquifers, river reaches, or any region that is part of the continent

## Observation Stations

- Ameriflux Towers (NASA & DOE)

- NOAA Automated Surface Observing System

- USGS National Water Information System

- NOAA Climate Reference Network

- Map for the US

**Notes:**
David  - for each of these site files, can you include how many points are in each one. 

## Inference Space

- “Transcending place” means testing hypotheses in areas thought to be similar (in some attributes).
- Digital continent will enable identification of “similar” areas and (some) data about that spot.
- Observatories will enable inference about similar regions (e.g., presumably one can infer more about Delaware R. from Potomac than about Rio Grande).

## DOs are the foundation of EOs

- Collaboration of Mission and Science Agencies
- DO contains both monitoring and research data
- DO supports hypothesis test, decision support systems, mgmt models
- Interdisciplinary communication
- Scientists can access multiple conceptualizations to improve understanding
- Everyone benefits from context provided
- Incentives must exist for people and agencies to *want* to contribute (and they do!)

## Water Resource Regions and HUC’s

## NHDPlus for Region 17E

## NHDPlus Reach Catchments  ~ 3km2

- About 1000 reach catchments in each 8-digit HUC

- Average reach length = 2km

- 2.3 million reaches for continental US

## Reach Attributes

- Slope
- Elevation
- Mean annual flow
- Corresponding velocity
- Drainage area
- % of upstream drainage area in different land uses
- Stream order

## HIS 1.0  (1 Nov 2006)

- Point Time Series
- Discovery and Publication
- Agencies
- USGS NWIS
- NCDC
- EPA Storet
- [LTER Trends]
- Static Federation to Observatory Test Beds

## CUAHSI Hydrologic Information System Architecture

- National HIS – San Diego Supercomputer Center

- Workgroup HIS – river authority, research centre or univ.

- Personal HIS – an individual scientist or manager

- HIS Server

- HIS Analyst

- Map interface, observations catalogs and web services for national data sources

- Map interface, observations catalogs and web services for regional data sources; observations databases and web services for individual investigator data

- Application templates and HydroObjects for direct ingestion of data into analysis environments: Excel, ArcGIS, Matlab, programming languages; MyDB for storage of analysis data

## HIS Server

- Supports data discovery, delivery and publication
- Data discovery – how do I find the data I want?
- Map interface and observations catalogs
- Data delivery – how do I acquire the data I want?
- Use web services or retrieve from local database
- Data Publication – how do I publish my observation data?
- Use Observations Data Model

## National and Workgroup HIS

- National HIS has a polygon
- in it marking the region of
- coverage of a workgroup HIS
- server

- Workgroup HIS has local
- observations catalogs for coverage of national data sources in its region.  These local catalogs are partitioned from the national observations catalogs.

*For HIS 1.0 the National and Workgroup HIS servers will not be dynamically connected.*

**National HIS**

**Workgroup HIS**

## CUAHSI Web Services

- NWIS

- ArcGIS

- Excel

- NCAR

- Unidata

- NASA

- Storet

- NCDC

- Ameriflux

- Matlab

- Access

- Java

- Fortran

- Visual Basic

- C/C++

- Some operational services

- Data Sources

- Applications

- Extract

- Transform

- Load

- _[http://](http://www.cuahsi.org/his)__[www.cuahsi.org](http://www.cuahsi.org/his)__[/his/](http://www.cuahsi.org/his)_

**Notes:**
Shauna – the ones on top are a bunch of web sites providing data that go through the CUAHSI Web Services (basically some software) which allows all this data to be used in all these other software packages listed across the bottom.  The ones in red are the ones that are connections that are already built.  It would look good to redo this in ESRI style.

## How Excel connects to ODM

**Excel**

**CUAHSI Web service**

**Obtains inputs for CUAHSI web methods from relevant cells.**
**Available Web methods are GetSiteInfo, GetVariableInfo GetValues methods.**

**converts standardized request to SQLquery.**

**imports VB object into Excel and graphs it**

**converts response to a standardized XML.**

**Observations **
**Data**
** ****Model**

***SQL query***

***Response***

**HydroObjects**

**converts XML to VB object**

**parses user inputs into  a standardized CUAHSI web method request.**

## Data Types

- Hydrologic
- Observation
- Data

- Geospatial
- Data

- Weather and Climate
- Data

- Remote Sensing
- Data

- (NetCDF)

- (GIS)

- (Relational database)

- (EOS-HDF)

- Digital
- Watershed

## HIS Extensions

- Integration of Weather Data
- Work with NCAR; prototype on Ohio
- Move from gridded to watershed-based delivery of data
- Hydrogeology
- Constructing stratigraphy for continent
- Geologic Framework
- Geomorphic and geologic history
- Incorporation of human dimension
- Transportation; structures
- Permits, Toxic Release Inventory, etc.
- Flood plain (contribution from real estate sector?)
- FEMA Lidar products
- Explicit development of AK, HI, PR beyond CONUS

## Digital Observatories

- Data Representation
- Need to develop integrated 4-dimensional data base
- Add coverages (easy) and fields (more complicated)
- Conceptual Frameworks and Modeling
- Can single data representation fulfill diverse set of science needs?
- Data assimilation techniques to guide sensor deployment and operation

## Environmental Observatories

- Two Paths:
- Bottom-up- Critical Zone Observatories (CZO). EAR- Hydrologic Science, Geochemistry/Biogeochemistry, Geomorphology
- Top-Down- WATERS Network- CLEANER/CUAHSI

## Observatory Development: Bottom-up

- EAR Research funds: CZO Solicitation
- Two sites selected in 2007
- CUAHSI will push for public data access
- If successful, these first two hopefully initial pieces of a prototype observatory network

## Observatory Development: Top Down

- Major Research Equipment and Facility Construction (MREFC)
- WATERS Network-
- Currently 11 testbeds have been either awarded or recommended for award

## WATERS Network

**An initiative of the U. S. National Science Foundation Engineering and Geosciences Directorates**

*HydroView*

**WATERS Network**

**Presenter Name, Affiliation**
- Presented at ....
- Date

## Environmental Observatories: a new approach for 

**Environmental Observatories: a new approach for **
**integrated, field-oriented collaborative research **
**at regional to continental scales**
- Rely on advances in:
-      **sensors and sensor networks** at intensively instrumented
-           sites shared by the research community
-      **cyberinfrastructure **with high bandwidth to connect the sites,
-           data repositories, and researchers into **collaboratories**
-      **distributed modeling** platforms

***From USGS***

## WATERS NetworkWATer and Environmental Research Systems Network

- Joint collaboration between the CLEANER Project Office and CUAHSI, Inc, sponsored by ENG & GEO Directorates at the National Science Foundation (NSF)
*CLEANER = Collaborative Large Scale Engineering Analysis Network for Environmental Research*
*CUAHSI = Consortium of Universities for the Advancement of Hydrologic Science*
- Planning underway to build a nationwide environmental observatory network using NSF’s Major Research Equipment and Facility Construction (MREFC) funding

## The Need...and Why Now?

*Nothing is more fundamental to life than water. Not only is water *
*a basic need, but adequate safe water underpins the nation’s *
*health, economy, security, and ecology.*
- 	NRC (2004) *Confronting the nation’s water problems: the role of research*.

- ● Water use globally will triple in the next two decades, leading to increases in erosion,
-    pollution, dewatering, and salinization.
- ● Only ~55% of US river and stream miles and acres of lakes and estuaries
-    fully meet their intended uses; ~45% are polluted, mostly from diffuse-source runoff.
- ● Polluted runoff caused more than 16,000 beach closings and swimming advisories in the
-    US in 2001
- ● Floods cause an average of $5.2 billion/year in damage and 80 deaths/year.
- ● Of 45,000 U.S. wells tested for pesticides, 5,500 had harmful levels of at least one.
- ● Fish consumption advisories are common in 30 states because of elevated mercury levels.

## Four critical deficiencies 

**in current abilities:**
-     1. Basic data about water-related
-         processes at the ***needed spatial ***
***        ******and temporal resolution***.
-     2. The means to integrate data across scales from different media and sources
-         (observations, experiments, simulations).
-     3. Sufficiently accurate modeling and decision-support tools to predict underlying
-         processes and forecast effects of different engineering management strategies.
-     4. The understanding of fundamental processes needed to transfer knowledge
-         and predictions across spatial and temporal scales—from the scale of
-         measurements to the scale of a desired management action

*Courtesy of Tom Harmon*

## DRAFT Vision for the WATERS Network: 

**DRAFT Vision for the WATERS Network:**** **

- WATERS is an integrated observing system that will transform our ability to predict the quality and quantity of the nation’s waters in real time.

## CLEANER GRAND CHALLENGE

**CLEANER GRAND CHALLENGE**

- How do we detect, predict, and manage the effects of human activities and natural perturbations on the quantity, distribution and quality of water in near real time?
-  Develop capacity to detect effects on water of key drivers:
-  Population and land use shifts
-  Energy, water and material resource use
-  Climate change

## CUAHSI GRAND CHALLENGE

**CUAHSI GRAND CHALLENGE**

- Develop a predictive understanding of continental water dynamics and its interaction with climate, landscape, ecology and civilization
- Three themes in CUAHSI science plan:
- Patterns and variability
- Climate and human development
- Prediction and adaptation to change

## The WATERS Network will: 

** **
**1. Consist of:**
-     (a) a national network of ***interacting field sites***;
-          across a range of spatial scales, climate and
-          land-use/cover conditions
-     (b) teams of ***investigators*** studying human-stressed
- 	     landscapes, with an emphasis on water problems;
-     (c) ***specialized support***** **personnel, facilities, and technology; and
-     (d) ***integrative cyberinfrastructure*** to provide a shared-use network
-          as the framework for collaborative analysis
** ****2. Transform **environmental engineering and hydrologic science research
-      and education
** ****3. Enable **more effective management of human-dominated environments
-      based on ***observation, experimentation, modeling, engineering ***
***     ******analysis, and design ***

## Common Vision: WATERS Network

- Sensors and Measurement Facility

- Synthesis

- Informatics

- Observatories/ Environmental Field Facilities

## Network Design Principles:

- Enable multi-scale, dynamic predictive modeling for water, sediment,
- and water quality (flux, flow paths, rates), including:
-         Near-real-time assimilation of data
-         Feedback for observatory design
-         Point- to national-scale prediction
- Network provides data sets and framework to test:
-       Sufficiency of the data
-       Alternative model conceptualizations
- Master Design Variables:
-        Scale
-        Climate (arid vs humid)
-        Coastal vs inland
-        Land use, land cover, population
-           density
-        Energy and materials/industry
-        Land form and geology

**Nested (where appropriate) Observatories **
**over Range of Scales:**
- Point
- Plot (100 m2)
- Subcatchment (2 km2)
- Catchment (10 km2) – single land use
- Watershed (100–10,000 km2) – mixed use
- Basin (10,000–100,000 km2)
- Continental** **

## I

- III

- II

- Simplified schema of a potential national **WATERS Network** based on three hydrologic
- regions: (I) coastal, (II) humid-continental, and (III) arid-continental; large blue circles:
- regional, watershed-based observatories; small circles: intensively instrumented field sites

## Humid Continental Watershed:

- Precipitation
- Deposition

- Agricultural Fields

- CAFOs

- Reservoir
-     Water Treatment

- Retention Basin

- Evaporation

- Potential WATERS EFF site

- Sensor

- Sensors

## Sensor Network

- Distributed network of interchangeable arrays of remote and embedded stationary and mobile sensors
- 	**Satellite and Remote Sensing**
-     **Individual Deployed Sensors**

## WATERS Network Cyberinfrastructure

**Potential Observatory **

**WATERS Network Cyberinfrastructure**

**Notes:**
QIT – fast development in the last 25 years. The configuration, operation and results that may  be provided by these techniques position them on the front-line of the state-of-the-art techniques.  They incorporate the most salient features of the new generation of instruments. They have become extensively used in fluid mechanics and thermodynamic investigations.  More recently QIT gain popularity in the civil engineering community.  My visit in Japan is aimed at continuing and boosting a Kobe Univ. – IIHR collaboration (started in 1995) in the area of creative implementation of QIT in civil engineering applications.
Imaged-based techniques for various applications have been developed at IIHR in the last six years.  These techniques are mainly implementations of fluid mechanics experimental tools for hydraulics research.

## WATERS Network CI Planning

- CLEANER Cyberinfrastructure Committee is creating a CI program plan with CUAHSI
- Two major projects are creating CI prototypes for WATERS (with more to come):
- CUAHSI Hydrologic Information System (HIS) project (David Maidment, PI)
- NCSA Environmental CI Demonstration (ECID) project (Barbara Minsker and Jim Myers co-leads)
- Have jointly proposed a draft common environmental CI architecture
- Are demonstrating how observatories can enable adaptive forecasting and management

## Environmental CI Architecture: Research Services

**Create Hypo-thesis**

**Obtain Data**

**Analyze Data &/or Assimilate into Model(s)**

**Link &/or Run Analyses &/or Model(s)**

**Discuss Results**

**Publish**

- Knowledge Services

- Data
- Services

- Workflows & Model Services

- Meta-Workflows

- Collaboration Services

- Digital Library

**Research Process**

**Supporting Technology**

- Integrated CI

**ECID Project Focus: Cyberenvironments**

**HIS Project Focus**

## Collaboration Services: CyberCollaboratory

- A web portal to allow sharing of information and ideas across the community.
- Currently being used by CLEANER Project Office team
- Public CI demo space shows numerous cyberinfrastructure demonstrations

- _[http://cleaner.ncsa.uiuc.edu/cybercollab](http://cleaner.ncsa.uiuc.edu/cybercollab)__ _

## CUAHSI Data Services

- Web Services
- Library

- Web application: Data Portal

- Your application
-  Excel, ArcGIS, Matlab
-  Fortran, C/C++, Visual Basic
-  Hydrologic model
-  ................

- Your operating system
-  Windows, Unix, Linux, Mac

- Internet

**S**imple **O**bject **A**ccess **P**rotocol

- Source: David Maidment, Univ. of Texas

## Path Forward: Timeline

**Path Forward: Timeline**

**2007  **Joint workshop on integrating social sciences into EOs (January)
- 	     Public comment period on draft program plan and conceptual design
- 	        (science, sensor, organization, social science, education, and
-               cyberinfrastructure-February)
**	****     **Completion of program plan and conceptual design (August)
**2008**  Consortium arrangements established
-           Conceptual Design Review - move to MREFC Readiness Status
**2009  **Continue funding development of enabling technologies, test-beds,
**             **and prototypes
**FY 2012  **Start of MREFC funding and construction phase
**FY 2016  **Project completion; start of full-scale operations

## Conclusions

- Networked observatories will create a new paradigm for environmental research
- Shared infrastructure at large scales
- Interdisciplinary teams collaborating remotely to address complex environmental issues
- Enabling improved understanding & management of large-scale natural environmental systems
- Cyberinfrastructure will be critical to the observatory’s goals
- Enabling real-time collaboration, data services, modeling, & management
- Creating a national knowledge network, not a set of individual field sites and investigators

## Acknowledgments

- 100s of researchers and educators from across the US are contributing to WATERS’ planning process
- _[http://cleaner.ncsa.uiuc.edu](http://cleaner.ncsa.uiuc.edu/)__ _
- _[http://www.cuahsi.org](http://www.cuahsi.org/)__ _
- Funding sources:
- NSF grants BES-0414259, BES-0533513, EAR-0413265, and SCI-0525308
- Office of Naval Research grant N00014-04-1-0437

## Questions?

- Contact info:
- Jon Duncan, Project Manager
- _[jduncan@cuahsi.org](mailto:jduncan@cuahsi.org)_
- 202 777 7305
- How can CUAHSI and CLEANER make better linkages with NWQMC?