Graduate Student Tests:
Evaluating the Usability of UFS
Cecelia DeLuca and Susan Jasko
for the UFS Communication & Outreach Cross-Cutting TeamFirst Annual UFS Users’ Workshop

July 27, 2020

What is the Graduate Student Test?
The Graduate Student Test (GST) assesses how easily a student can:
●
●
●
●
●
●
●

Get code.
Run code.
Change code.
Tes t code for correct operation.
Evaluate code with s tandard diagnos tic packages .
Get docum entation, us er s upport, and training.
Unders tand what is needed for their code changes to trans ition to operations .
There can be different GSTs for different applications , and each tes t m ay cover s ubs etsof the tas ks above.

You don’t need to be a graduate student to take a Graduate Student Test!

Motivation● Improve the usability of UFS application releases by collecting, analyzing,and responding to feedback● Create an easy way for community members to try running UFS
applications - encourage graduate students to use UFS code in their work!
● Develop a body of evidence about the efforts of the UFS project to workwith the community and document its successes and challenges

Read about the origins of the Graduate Student Test in the Spring 2020 UFS newsletter:
https://us19.campaign-archive.com/?u=76c42eac2d7040bc5fe22c8d0&id=1312b944fa

Approach● Base tests on well-defined sequences of tasks○ Experiments and tasks developed in collaboration with UFS releaseteams● Collect feedback using a questionnaire○ Developed in collaboration with communication specialists and social
scientists on the UFS Communication & Outreach Cross-Cutting Team● Limit time required from participants to less than a day

Graduate Student Tests Deployed & Plan
●GST1 (completed in fall 2019): UFS S2S Prototype SST Experiment– Run a na tm os phere/ la nd/ ocea n/ ice coupled s ys tem for 5 da ys , m a ke a code cha nge to m odify the SSTs ent to other com ponents by the ocea n, rerun a nd com pa re res ults . Includes a porta ble workflow. 6hours a llotted.

●GST2 (open): UFS Weather Model Cloud Condensation Nuclei Experiment– Run the UFS wea therm odel for 24 hours , cha nge cloud condens a tion nucleii na m elis t pa ra m eters , rerun a nd com pa re.
Does not include a workflow. 3 hours a llotted.

●GST3 (open): UFS Medium-Range Weather (MRW) Application Cloud Condensation Nuclei
Experiment – Run the MRW Application for 24 hours, change cloud condensation nucleii namelist
parameters, rerun and compare results. Includes a portable workflow. 6 hours allotted.

●GST4 (upcom ing in fa ll 2020): UFS Short-Ra nge Wea ther Applica tion - TBD

Overall Graduate Student Test Results

So far, there have been 14 Graduate Student Test responses

MRW Graduate Student Test Results (1)MRW Graduate Student Test Results (2)MRW Graduate Student Test Results (3)stronglyagree

I needed additional documentation in order to get started.

agree

neutraldisagreestrongly
disagree

1

1

3

2

1

I found it easy to understand the configuration being modeled.

2

4

I found the code easy to get.

52

I was able to run the code without any trouble.

3

1

3I found the configuration easy to modify.

4

21

MRW Graduate Student Test Results (4)Responses to: Is there anything else you would like us to know?
●You folks a re a wes om e! Keep up the grea t work!

●A very good firs t s tep in the right direction. Ma jor Kudos to the entire UFS tea m on getting it this fa r.
Congra tula tions !

●Overa ll, I thought it wa s ea s y to us e for s om eone like m e who ha s experience running clim a tem odels .

●Grea t job. The AMS webina r wa s very helpful a s well. Tha nks .

●This 1.0.0 relea s e is a grea t s tep! Very im pres s ed tha t I could run it on m y la ptop. It lea ves m ecra ving m ore docum enta tion of the knobs tha t ca n be turned.

●Tha nks for being pa tient with m e ta king s o long to get this com pleted. With COVID-19 m y s chedulea nd routine got m es s y...gla d I wa s a ble to finis h this a nd hope to lea rn m ore a bout the UFS a s it
develops !

Try a Graduate Student Test!
To learn more, view previous responses, and register for a Graduate Student Test:
https://ufscommunity.org/science/gst/

UFS Communication and Outreach Team
Susan Jasco, University of Alabama
Cecelia DeLuca, CIRES
Bhavana Rakesh, STC/STI Modeling
Jeff McQueen, NOAA EMC
Sarah Lu, University of Albany/NCAR
Mike Ek, Joint Numerical Testbed/RAL
Lauren Gaches, NOAA Communications
Eric Rogers, NOAA EMC
Jennifer Folta-Teitelbaum, STI Modeling
Cristiana Stan, GMU

Hendrik Tolman, NOAA NWS
Richard Rood, University of Michigan
Jimy Dudhia, NCAR
Douglas Hilderbrand, NOAA NWS Office ofCommunication
Gina Eosco, NOAA WPO
Maoyi Huang, STI Modeling
Yan Xue, STI Modeling
Tim Schneider, NCAR RAL
Jennifer Sprague, NOAA