Instructions for data analysis software:- Preparation- The IUGONET project and its products for space weather study- Installation- How to Use SPEDAS, part1- How to Use SPEDAS, part2
Published by IUGONET Project Team, Sep. 2017.
http://www.iugonet.org/?lang=en

IUGONET System: MetaData System for Space
Weather and Earth Observation Data Analysis
Shuji Abe
International Center for Space Weather Science and Education

Outline

Today’s Outline:
This hands-on have 4 topics
1.
2.
3.
4.

Introduction to the IUGONET
Analysis software(SPEDAS) hands-on 1
loading and plotting built-in data
break
Analysis software(SPEDAS) hands-on 2
figure reformation and data processing
Analysis software(SPEDAS) hands-on 3
loading and plotting external data

Characteristics of Upper Atmosphere

1.
2.
3.

Affected by various phenomena from the earth surface to the space
Many physical parameters
Various variations including solar activity are overlappedGround Observations for Upper Atmosphere

JpGU 2014

Objectives of the IUGONET
Problem

SolutionIUGONET provides a new research platform thatenables metadata extracted from ground-based
observation data to be shared.
In addition, IUGONET developed analysis
software to access and analyze data in an
integrated fashion.

Observational data should be quality
controlled and managed by the specialists
who know the observations.
For users….It was not easy to reach anecessary information, since databases
are distributed in various universities andinstitutes.

Overview of the project
The Inter-university Upper atmosphere Global Observation
NETwork (IUGONET) project aims at establishing “einfrastructure” for researchers to effectively find, get, andanalyze various kinds of upper atmospheric data spreadover Japanese universities and institutes.
 To exchange ground-based observation data accumulated over
50 years since IGY (both digital and analogue data)

Planetary Plasma andAtmospheric ResearchCenter To promote analyses of multi-disciplinary data, which will lead tocomprehensive studies of mechanisms of long-term variations in
the upper atmosphere
WDS for Geomagnetism
International Center forSpace Weather Scienceand Education

Kyushu University

Tohoku University

Kwasan and Hida
Observatories

Kyoto University
WDS for Aurora

Data Analysis Center for
Geomagnetism and Space
Magnetism

National Institute ofPolar Research

Kyoto University

Solar Terrestrial Environment
Laboratory
Research Institute for
Sustainable HumanosphereKyoto University

Nagoya University
WDS for Cosmic Rays

7

©2011 Google - Map data ©2011 Geocentre Consulting, ZENRIN, Europa Technologies, Mapabc, SK M&C -Schematics of the projectInternational
Users

Expand the system to other
Geoscience community

Kyushu Univ.
ICSWSE
NagoyaUniv.
STEL

Developers

MD

MD

WDC for Cosmic
Rays

IUGONET
Information Center

MD

MD

TohokuUniv.
PPARC

Kyoto Univ.

MD
MD

NIPR
WDC for Aurora

Chairperson

MD
WDC forgeomagnetism Space
Magnetism

Kyoto Univ.
RISH

Kyoto Univ.
DACGSM

Astronomical
Observatories

IUGONET Metadata Format/Structure
IUGONET metadata format = SPASE + modifications(http://www.spase-group.org)

What’s SPASE?

metadata format developed by international
consortium to comprehensively describe
research resources regarding heliospheric andmagnetospheric satellite observations

 closely related to STP and upper atmosphereresearches ( easy to use as a base format)
 new metadata elements & words appendable (customizable according to our data ) widely-used in VxOs ( possible metadata
exchange in the future)

IUGONET’s modifications
 additional words to represent non-digital archives
 additional words to represent heliospheric coordinates new metadata elements to describe observation location & range

IUGONET Metadata Database

http://search.iugonet.org/
 IUGONET MDB (called IUGONET Type-A) is capable of cross-searchingobservational data distributed across the IUGONET institutions.
 IUGONET Type-A brings a remarkable advancement in accessibility to theobservational data and accelerate the interdisciplinary study.
 IUGONET Type-A provides a one-stop web services such as
searching data, finding interesting events, interactively plotting the data,
and leading users to more detailed analysis.

Analysis Software SPEDAS• The IUGONET Data Analysis Software(UDAS) is the plug-in software for Space
Physics Environment Data Analysis
System(SPEDAS), formerly known as
THEMIS Data Analysis Software suite (TDAS)• The IUGONET data (e.g., geomagnetic data,
aurora data, radar data, and so forth) andmany satellite mission data (THEMIS, GOES,
WIND, and ACE) can be handled.

SPEDAS/TDAS• It is possible to use many routines tovisualize and analyze time series data.
• It accesses the IUGONET data through theInternet, and then the data areautomatically downloaded onto the user's
computer

Relationship between
UDAS, SPEDAS, and IDL

Outline of Loading/Plotting Data Using SPEDAS
Data Servers on
the Internet

User PC
Automatic
download
Data areloaded as
tplot variables

data

SSL, Berkeley,
THEMIS, GBO
CDAWeb,OMNI, ACE,
Wind, etc.

Directories for the
downloaded data arecreated automatically.

data
data

Data can be easily plotted, for example, by only three basis commands
with the SPEDAS-CUI tool.
1. Set a time period
2. Load *** data
3. Plot the loaded datatimespan, ‘yyyy-mm-dd’
iug_load_***tplot, +++If using the GUI tool, only a few simple clicks of your mouse are required to make thesame plot as that created by the above command with the CUI tool

Examples of SPEDAS visualization
Time series stack plot

Correlation

Mapping

WaveletContributions for Space weather study
Data supported by SPEDASSun

Magnetosphere

Satellite data
Stereo SOHO

Wind

ACE

IMP-8

THEMIS
Satellite

LANL

NASA
OMNI

ERG

GOES

VAP Satellite
EFW
RBSPICEGround-based observational data
THEMIS
Geomag.

THEMIS
CameraCARISMA
Geomag.

GIMA
Geomag.

Greenland
Geomag.

MACCS
Geomag.

USGS
Geomag.

IUGONET, ERG

Solar Telescope, Solar and planetary radio
telescope, Ionosphere radar (SuperDARN,EISCAT, etc.), Atmosphere radar (MU, EAR,etc.), Meteorological observation data,
Geomag. network（WDC, MAGDAS,
210MM, Antarctica・Iceland, etc.）

Solar
Telescope

THEMIS Satellite

Atmosphere& Ionosphere

Interdisciplinary study
 Many missions have provided plugins for SPEDAS.
 IUGONET has also provided a plugin for SPEDAS, which includes manyroutines for loading various ground-based observation data.
 SPEDAS is suitable for Space Weather study.

Summary and Future Vision
NGDC
…

NSSDC
…

Big Data
Centers

VHO

VMO

Middle Class DC

VEPO
VITMO

Virtual Obervatory(US STP community)

VSPO

ViRBO

(Project of EU STP
community)

Various Geosciences
Databases in JapanICSU - WDS
ICSU - CODATA(International Framework)

Japanese STP
community

IUGONET is a basis of a network
datacenter

IUGONET includes three
WDC members-> Functioning like a big data
center efficiently

・Kyoto Univ.(Geomagnetism)
In progress
・STEL(Cosmic Ray)・NIPR(Aurora)Hand on of SPEDASPreparation for hands on
Prepare 64 bit Operating System.
1. Access the following URL
https://archive.iii.kyushu-u.ac.jp/public/PW4ogAoJc0AApxsBi0tkoyj6RAoFJuLftBc5j4JDfbsO

2. Download SPEDAS 3.1 zip file for your operating system(Win orMac), and then unzip it to your desktop.
spedas_3_1_win64_85_105_beta.zip or spedas_3_1_mac64_85_105_beta.zip
3. Download data.zip file including data for this hands-on, and thenextract it to the following directory.
 Windows: C:/data
 MAC: /Users/(username)/data
Troubleshooting.pdf and SPEDAS
wiki(http://spedas.org/wiki/index.php?title=Downloads_and_Installation) will
help you against some installation problems.

Preparation for hands on
4. In section 3, you can load and plot your own data on SPEDAS. Please
prepare it with the following format.
Supprted format:
1. CDF（Common Data Format）
2. Ascii
In this hands-on two format types shown are supported:0) Time series data arranged in the following order (i.e., date, time, and data);----------------------------------------------------date[0] time[0] ydata1[0] ydata2[0] ydata3[0] …
date[0] time[1] ydata1[1] ydata2[1] ydata3[1] …
date[0] time[2] ydata1[2] ydata2[2] ydata3[2] …
date[0] time[3] ydata1[3] ydata2[3] ydata3[3] …
date[0] time[4] ydata1[4] ydata2[4] ydata3[4] …
----------------------------------------------------where, ydata1, ydata2, ydata3, … are the column data.
As for the date[] and time[] format string, various formats are acceptable, for example,
YYYY-MM-DD/hh:mm:ss
yy MM DD hh mm ss
hh mm ss
See example “data/testfile_format0.txt”

Preparation for hands on1) Time series data that includes more than one row data at the same time----------------------------------------------------date[0] time[0] vdata[0] ydata1[0] ydata2[0] ydata3[0] …
date[0] time[0] vdata[1] ydata1[1] ydata2[1] ydata3[1] …
:date[0] time[0] vdata[99] ydata1[99] ydata2[99] ydata3[99] …
date[0] time[1] vdata[0] ydata1[100] ydata2[100] ydata3[100] …
date[0] time[1] vdata[1] ydata1[101] ydata2[101] ydata3[101] …
:
date[0] time[1] vdata[99] ydata1[199] ydata2[199] ydata3[199]
where vdata repeats every time, for example, altitude for the atmospheredata,frequency for the spectral data, and range for the radar data.
See example “data/testfile_format1.txt”

For more details, let’s see

http://www.iugonet.org/data/udas/How2_use_egg_en.pdf

Start of IDL-VM(GUI) tool
[1] Unzip the zipped SPEDAS file.
[2] Double-click the executable file
named ‘spedas’ in the directory‘spedas_v_3/spd_gui’.

Doule-click the executable
file named ‘spedas’

Click the icon
‘spd_gui’.

[3] IDL Virtual Machine window opens on
your PC, so please click the ‘spd_gui’button.

For Windows 10 users
If you encountered any graphics problem on
Windows 10, try launching IDL in"Windows 8 Compatibility Mode" orsomething similar. To do that:
0. Right click on the IDL or SPEDAS
executable file and select "Properties".
1. Select "Compatibility" tab
2. Check "Run the program incompatibility mode for:"3. Select "Windows 8" or "Windows 7".
Click ok the apply the changes.
Restart the program and then try your plotover again.

21

Start of IDL-VM(GUI) tool

Ready?

How to Use SPEDAS
part1
・Load data
・Plot data
・Save figure, data, and your work

GUI Basic Operation
Lesson:Load Dst index
1. Click this icon

Or,
Select “Load Data” from “File” dropdown menu.

GUI Basic Operation
1. Click IUGONET Tab

2. Uncheck “Use Single Day”

4. Chenge Instrument Typegeomagnetic_field_index

3. Set Date and Time
Start Time: 2012-03-04/00:00:00
Stop Time: 2012-03-11/00:00:00

GUI Basic Operation

2. Click the arrow

1. Choose three parameters- Dst_index- *(all)- *GUI Basic Operation

Display of Data Use Policy

1. Click“OK”

GUI Basic Operation

1. Data was loaded successfully!

2. Click“Done”

GUI Basic Operation
Lesson:Plot data
1. Click this icon

Or,
Select “Plot/Layout Options…” from “Graph”
dropdown menu.

GUI Basic Operation

2. Click “Line”

1. Select data which you want to plot:
wdc_mag_dst_prov

GUI Basic Operation

1. Selected variable nameis added to this box

2. Click OK

GUI Basic Operation
PanelX(Y,Z)
X: panel serial number
Y: row index of the panel
Z: column index of the panel

You can add, remove, and edit
panels with these buttons.

You can move panels
and change the numberof panels per page.

Lock to panel:
Lock panel axes tocurrently selected panel.

GUI Basic Operation
Try:
Load other three data
AE index(in IUGONET tab)
Instrument: magnetometer
MAGDAS magnetometer(in IUGONET tab)
Instrument: magnetometer
Station: HER
MAGDAS magnetometer(in IUGONET tab)
Instrument: magnetometer
Station: ASB

GUI Basic Operation
Answer

1. Select IUGONET tab

2. Select magnetic_field_index
4. Click arrow
3. Select AE_index, *(all), *5. Data is loaded

GUI Basic Operation
Answer
1. Select geomagnetic_field_fluxgate
3. Click arrow
4. Data is loaded
2. Select magdas#, asb & her, *5. Click DoneGUI Basic Operation
Lesson:Add Plot
wdc_mag_ae_prov_1min

1. Click this icon

Or,
Select “Plot/Layout Options…” from “Graph”
dropdown menu.

GUI Basic Operation

1. Click Add

3. Click “Line”
4. Data are added
2. Select wdc_mag_ae_prov_1min

5. Click OK

GUI Basic Operation
1. Click black triangles

GUI Basic Operation
Try:
Plot other two data
magdas_mag_her_1sec_hdz

magdas_mag_asb_1sec_hdz

GUI Basic Operation
Answer

2. Click Line

3. Data are added
1. Select magdas_mag_her_1sec_hdz

GUI Basic Operation
Answer

2. Click Line

1. Select magdas_mag_asb_1sec_hdz

4. Click OK

3. Data are added

GUI Basic Operation

GUI Basic Operation
Lesson:Save plot as figure
1. Select
File – Export To Image File

2. Select save folder

2. Input file name and select format (by extension)

3. Click “save”

GUI Basic Operation
Confirmation

Back to previous

Click “Save”

GUI Basic Operation

X pixel 856, Y pixel 554(65%)

X pixel 428, Y pixel 554

GUI Basic Operation
Lesson:Save data as ascii
1. Select
File – Save Data As

GUI Basic Operation
2. check this box
3. Select time interval
1. Select data which you want to savemagdas_mag_her_1sec_hdz_x
4. check this box

6. Select save folder

7. Input file name (data is saved in csv format)
5. Click Save
8. Click “save”

GUI Basic Operation

An ascii data file was
successfully saved!!!

GUI Basic Operation
Lesson:Dump your workspace
1. Select
File – Save SPEDAS Document

2. Select save folder

3. Input file name

4. Click “save”※ SPEDAS Document is written in XML format

Coffee Break…

How to Use SPEDAS
part2
・Restore your work
・Manage axis
・Process and data

GUI Basic Operation
Lesson:Restore part1 workspace

1. Exit SPEDAS
2. Run SPEDAS again
3. Select File- Open SPEDAS Document
4. Click “Yes”
5. Select the saved tgd file.

GUI Basic Operation
Lesson:Remove plot

1. Select
Graph – Plot/Layout Options

GUI Basic Operation
2. Click “Remove
1. Select
wdc_mag_ae_prov_1min_time -vs- wdc_mag_ae_prov_1min_4
in the right-hand panel.

3. Remove
wdc_mag_ae_prov_1min_time -vs- wdc_mag_ae_prov_1min_3
wdc_mag_ae_prov_1min_time -vs- wdc_mag_ae_prov_1min_2
wdc_mag_ae_prov_1min_time -vs- wdc_mag_ae_prov_1min_1
in the same way

4. Click OK

GUI Basic Operation
ResultGUI Basic Operation
Lesson:Change X range (time scale) of
the plot (1)Reduces X range bymajor tick marker
Expands X range bymajor tick marker
Shift left X range bymajor tick marker
Shift right X range
by major tick marker

Use these icons

GUI Basic Operation
Lesson:Change X range (time scale) of the plot (2)1. Select
Graph – X Axis Options

GUI Basic Operation
1. Select Panel (If panel is locked,
use “Apply to All Panels”.)
2. Select Fixed Range

3. Change values
Min 2012-03-06/00:00:00.000
Max 2012-03-11/00:00:00.000

4. Click "Apply to All Panels”

GUI Basic Operation
Lesson:Customize Ticks.

1. Select Major Ticks By
Number or Interval.
2. Input the number into
# (Number or Interval)of Minor Ticks

GUI Basic Operation
Lesson:Change Annotations

1. Select your favorite
format in the pull-downmenu of Annotation Format.

2. If you want tochange the character
font, size, and color,select your favorite
format in the pull-downmenu here.

3. Click "Apply to All Panels”

60GUI Basic Operation
Lesson:Customize Labels (of X axis)

1. Select Panel 4 (bottom
panel)

2. Check the“Show Label”
box

3. Type “Universal Time ”
on the Edit/Add Label
4. Click "OK“.

GUI Basic Operation
ResultGUI Basic Operation
Other options.
• Page Options…
Customize the text and layout of thepage.
• Panel Options…
Customize the title and color of eachpanel.
• Line Options…
Customize the line and symbol of eachplot panel.
• Legend Options…
Customize the legend which appears
when you put the mouse cursor on theplot.
• Variable Options…
Display the values of the selectedparameters under the time label.

GUI Basic Operation
Lesson:Reset X range (time scale)
1. Select X Axis Options

GUI Basic Operation
1. Select (L) Panel 1(1, 1) 2. Select Auto Range

3. Click “OK”

GUI Basic Operation
Lesson:Processing- subtract average

1. SelectAnalysis – Data Processing

GUI Basic Operation
4. Click Subtract Average

3. Active Data are added
2. Click right arrow

1. Select data you want to process
magdas_mag_asb_1sec_hdz
magdas_mag_her_1sec_hdz

GUI Basic Operation

New variables are created.
magdas_mag_asb_1sec_hdz-d
magdas_mag_her_1sec_hdz-d

1. Click DoneGUI Basic Operation
Open “Plot/Layout Options”

1. Remove Panel 3 and 4

3. Click line
4. Data are added

2. Selectmagdas_mag_her_1sec_hdz-d

5. Then, add the other variable,
magdas_mag_asb_1sec_hdz-d
to panel in the same way.

6. Click OK

GUI Basic Operation

Subtracted average!

GUI Basic Operation
Try:
Plot Power Spectrum of
magdas_mag_her_1sec_hdz_x

Hint1: Use “Data Processing” forcalculating (if you will get an
option dialog, use default value)
Hint2: Use “Spec” for plotting

1. SelectAnalysis – Data Processing

GUI Basic Operation

2. Active Data is removed

1. Click Clear Active

GUI Basic Operation

3. Active Data are added
2. Click right arrow

1. Select data
magdas_mag_her_1sec_hdz

5. Click OK

4. Click Power Spectrum

GUI Basic Operation

1. New variables are created!

2. Click DoneGUI Basic Operation
Open “Plot/Layout Options”

2. Click Spec

3. Data are added

1. Selectmagdas_mag_her_1sec_hdz_x_dpwrspc

4. Click OK

GUI Basic Operation
ResultThis color bar can becustomized inGraph - Z Axis Options.

GUI Basic Operation
Lesson:Use Calculate (Equation editor)
1. SelectAnalysis – Calculate…

GUI Basic Operation
Equation Editor for SPEDAS
Variable=Your loaded data

Built-in function

Built-in Operator

Built-in Constant

GUI Basic Operation
Lesson:Make an equation using the loaded variables.

Type variable/function/Operator/Constant, and make equation
A = B + C - D…
Note:
Enclose the tplot variable in double quotation marks

GUI Basic Operation
Lesson:Make an equation using the loaded variables.
1. Select magdas_mag_her_1sec_hdz-d_x
3. Variable is added
2. Click arrow

Then, try to add the offset (+200) to
magdas_mag_her_1sec_hdz-d_x
and plot on new panel.

GUI Basic Operation
Note: one line, never return

Answer"magdas_mag_her_1sec_hdz-d_x_ofst" = "magdas_mag_her_1sec_hdz-d_x" + 200

1. Click Run
2. A new variable is created
3. Click done

GUI Basic Operation
Open “Plot/Layout Options”

1. Select Panel 3 and Remove it

3. Click Line

4. Data are added

2. Selectmagdas_mag_her_1sec_hdz-d_y
magdas_mag_her_1sec_hdz-d_x_ofst

GUI Basic Operation
Open “Plot/Layout Options”

1. Change the value of Row to 3

2. Panel 6 is changed to (3, 1)

3. Click OK

GUI Basic Operation
ResultOffset (+200nT) was
added to the H comp.

GUI Basic Operation
Try:
Expand the plot using the mouse.
By left-click anddrag the mouse

A new page opens

How to Use SPEDAS
part3
・Additional data loading

GUI Basic Operation
Lesson:Load Additional ASCII Data

Sample1: magnetometer data

HEADER(13 lines)

Data(86400lines)

See Data/testfile_format0.txt

Data(6 columns)

GUI Basic Operation

1. Select
File – Load Your Data – Load ASCIIGUI Basic Operation
1. Click”Browse”, and select‘testfile_format0.txt’.
2. Format Type: Select 0
3. Time Format: Check Specify, andput ‘YYYY-MM-DD hh:mm:ss.fff’
4. Column No. of loaded data:put ‘1,2,3,4’Note:Column number starts from 0.

5. Options for Header:
Check the box, and put ‘13’ toNumber of lines of skip.

6. Click OK

GUI Basic Operation

1. Click “OK”

GUI Basic Operation
Open “Plot/Layout Options”

2. Click Line

1. Click +, and selecttvar1_0

3. Data are added

4. Repeat the same process totvar1_1, tvar_1_2, and tvar1_3

5. Click “OK”

GUI Basic Operation

Click this icon before next lesson

Magnetometer data
written in
ASCII(IAGA-2002)
format are plotted.

GUI Basic Operation
Lesson:Load External ASCII Data

Sample2: EISCAT radar data

HEADER(starts from %)Data(11520lines)

See Data/testfile_format1.txt

Data(19 columns)

GUI Basic Operation
Open File – Load Your Data – Load ASCII

1. Click”Browse”, and select‘testfile_format1.txt’.
2. Format Type: Select 1
3. Time Format: Check Specify, andput ‘YYYY-MM-DD hh:mm:ss.f’
4. Column No. of loaded data:put ‘5,6,7,8’
5. Loaded data name:put ‘Ne, Vi, Ti, Te’
6. Column No. of v_vector:
put ‘1’

5. Click OK

7. Options for Header:
Check the box, and put ‘%’ toComment symbolGUI Basic Operation

1. Click “OK”

GUI Basic Operation
Open “Plot/Layout Options”

2. Click Spec

3. Data are added
1. Select Ne
4. Repeat the same process to
Vi, Ti, and Te

5. Click “OK”

GUI Basic Operation

Click this icon before next lesson

EISCAT radar data written in
ASCII format are plotted inspectrogram.

GUI Basic Operation
Try:
Plot your own ASCII/CDF
data using File – Load Your
Data – Load ASCII

If you do not prepare yourown data, let’s try to plot
Data/practice.txt
Hint: Use format 1 for dataloading, and use SPEC fordata plotting

GUI Basic Operation
Answer
1. Click”Browse”, and select‘practice.txt’.

Open “Load ASCII”

2. Format Type: Select 1
3. Time Format: Check Specify, andput ‘YYYY-MM-DD/hh:mm:ss.fff’
4. Column No. of loaded data:put ‘1,2,3’
5. Loaded data name:put ‘u, v, w’
6. Delimiter: put ‘,’
7. Column No. of v_vector:
put ‘0’9. Click OK

8. Options for Header:
Check the box, and put ‘#’ to Comment
symbolGUI Basic Operation
Open “Plot/Layout Options”

1. Select u

3. Data are added
2. Click Spec

4. Repeat the same process tov and w

Practice data(wind velocity
observed by MU radar) written
in ASCII format are plotted inspectrogram.

For advance…
UDAS website：http://www.iugonet.org/product/analysis.jsp

Acknowledgment
SPEDAS is a grass-roots data analysis software for the Space Physics community, which wasdeveloped by scientists and programmers of the UC Berkeley's Space Sciences Laboratory,
UCLA's IGPP and other contributors

http://www.iugonet.org/

Feedbacks
If you have any feedbacks, questions, requests about thishands-on and software, please send email to the following:

Subject: ICeSSAT2018 SPEDAS hands-on
To: abeshu@icswse.kyushu-u.ac.jp

We would be appreciated your many comments!