SuperAI on Supercomputers
Haohuan Fu
haohuan@tsinghua.edu.cn
High Performance Geo-Computing (HPGC) Group
http://www.thuhpgc.org

Earth Science and Supercomputersn Create a digital earth,so as to:p simulate
p analyze
p understandp predict and mitigate

figure credit: Earth Simulator, JPN

Two Major Functions

Simulation

Data Analysis
3Two Major Functions

To design highly efficient andhighly scalable simulation
applications

To develop intelligent data
mining methods for theanalysis of BIG scientific DATA

4

Two Major Functions

To design highly efficient andhighly scalable simulation
applications

To develop intelligent data
mining methods for theanalysis of BIG scientific DATA52015-now: The CESM Project on Sunway TaihuLight

CAM5.0POP2.0CPL7
CICE4.0

CLM4.0

CESM1.2.0
Tsinghua + BNU 30+ Professors and Students• Four component models, millions lines of code
• Large-scale run on Sunway TaihuLight• 24,000 MPI processes• Over one million cores• 10-20x speedup for kernels• 2-3x speedup for the entire model

Haohuan Fu, Junfeng Liao, Wei Xue, Lanning Wang and et al., “Refactoring and Optimizing the Community
Atmosphere Model (CAM) on the Sunway TaihuLight Supercomputer”, The International Conference for High
Performance Computing, Networking, Storage and Analysis (SC), Salt Lake City, Utah, US, 20166

2017: Non-Linear Earthquake Simulation
Dynamic Rupture Source Generator(Based on CG-FDM)
Fault StressFriction
Wave Eqn
Init
Law Ctrl
Solver

Source Partitioner
Seismic Wave Propagation(Based on AWP-ODC)
VelocityStress Update
UpdateNext Timestep
StressAdjustment For
Plasticity

SourceInjection

3D Vel/Den Model

3D Model Interpolator

Snapshot/Sesimo
RecorderRestartController

LZ4 Compression, Group I/O, Balanced I/O Forwarding

2018: Simulating the Wenchuan Earthquake withAccurate Surface Topography(a)SurfaceTopography(b)Sunway (2017)
TangshanSunway (2018)
Wenchuan

5kmz'

y−T-&−T.

! = !($, &, ')) = )($, &, ')* = *($, &, ')T.
/& = 0

T-

$x
(1)

(2)

Two Major Functions

To design highly efficient andhighly scalable simulation
applications

To develop intelligent data
mining methods for theanalysis of BIG scientific DATA

9

Data Challenge for Climate Scientists• Over 100 PB just forclimate change studies• A Challenge but also
a huge opportunity
Climate Data Challenges in the 21st Century,
Overpeck et al., Science , 331 (6018): 700-702

Remote sensing dataSatellite

Google earth images
1984-2018

UAV

UAV image- Recording what is happening on the earth for decades- What issues can we solve based on these data?

High-resolution image

11

Look Ahead: Data-Driven Modeling and Prediction

migrationof birds

city
green land

ecologyprotection
zone

Land
Cover
Mapping

waterbody

RemoteSensing
Data Sets              

sea-level  

Deep learning in computer vision
Image Classification

CatObject Detection

Semantic Segmentation
SkyTrees
CatGrasses
13

Deep learning + remote sensing dataChallenges:- Few public labeleddatasets in RS domain- Difference between RS
images and CV images- Real-world applications(small size, imbalance)
More accurate
land cover mapping

More intelligent
oil palm monitoring

More reliableurban land use mapping
Major issues:- How to select andbuild the RS datasets?
- How to select themost suitable method?
- How to improve themethod for our tasks?
14

Case 1: More Accurate Land Cover Map

First 30 m resolution global land cover maps

Running SVM on 8900 Landsat scenes on a supercomputer, achieved result in one day.

Gong et al., 2013, IJRSFROM-GLC (Accuracy: 63.72%)

Gong et al., 2013. IJRSA Starting Point: Direct Application of Stacked AutoEncoder

Landsat Image

RF Image

SAE ImageRFSVM

ANNSAE

Overall Accuracy

76.03%

77.74%

77.86%

78.99%

Mapping Time

33.605 ± 0.183

16344.188

4.014 ± 0.003

13.250 ± 0.042
18Integrating Google Earth Image
Spatial features from 0.5-m images

Google EarthPredicted
Result 1CNN

SVM

Predicted
Result 2

SVM

Predicted
Result 3

RFPredicted
Result 4

Spectral features from 30-m images
Surface reflectance, NDVI, DOY
Longitude and Latitude
Elevation and Slope
Surface reflectance of max NDVI, DOY

Land cover map

24
Landsat + DEM
19

Integrating Google Earth Image

20

Integrating Google Earth Image

21

Integrating Google Earth Image

22

Integrating Google Earth Image

Slight increase using
30-meter resolution images

Great increase using
Multi-resolution images

23

Integrating Google Earth Image
Google Earth image

Results of RF

Results of SVM

Results of Ours

LegendImperviousForestCroplandWaterGrassland
Bare land
Shrubland
Cloud

Fewer confusions among various land cover types in the results of our proposed method.

2430m to 10m

Gong P., et al., 2019. Stable classification with limited sample: transferring a 30-m resolution sample set
25
collected in 2015 to mapping 10-m resolution global land cover in 2017,Science Bulletin.

10m to 3m

26

削弱部分时相或局部区域厚云遮盖影像City of Changsha

Case 2: Detection of Oil Palm Trees

Case 2: Intelligent monitoring of oil palm trees
Mapping of oil palm trees using
high-resolution satellite images

Detection of oil palm trees using
high-resolution satellite images

Detection and classification ofoil palm trees using UAV images

29

CNN based large-scale oil palm tree detection

30

Semantic segmentation based oil palm plantation extraction• We proposed the first semantic segmentation based approach for large-scale oil palm plantation extraction from
QuickBird Satellite images in 0.6-m spatial resolution.
• The enhanced pixel-wise dataset contains 36,000 images in 256×256 pixels located in southern Malaysia, including
four categories of samples: oil palm, other vegetation, buildings and the others.
• We present an end-to-end deep convolutional neural network based on the SegNet model, which has a symmetricalencoder-decoder architecture based on the convolutional layers of the VGG-16 model.
• Our proposed approach combines the SegNet model with the Conditional Random Fields and integrates the postprocessing results with the outputs of SegNet to improve the localization of boundaries.

Fully connected
CRF

Input

Encoder

conv-BN-ReLU-poolingconv+BN+ReLU

pooling

upsampling

Softmax

Decoder

Segmentation

upsamling-conv-BN-ReLU

31

CNN based large-scale oil palm tree detection
Multi-level CNN training and optimization•••

The first CNN is used for land cover classification to locate the oil palm plantation area, including three types of samples (oil palm plantationarea, other vegetation / bare land, and impervious/cloud).
The second CNN is used for object classification to identify the oil palms, including for types of samples (oil palm, background, othervegetation / bare land, and impervious/cloud).
The two CNNs are trained and optimized independently based on 17,000 training samples and 3000 validation samples.

CNN-1: Land cover classification

CNN-2: Object classification32

Transfer Learning
One Source Domain to One Target Domain

One Source Domain to Multi Target Domain

Case 3: Urban Land Use Map

Case 3: More accurate urban land use map• A building extraction method based onthe U-Net semantic segmentation model.

GIS Map data

Satellite imagery

Predicted buildings• A data fusion method combining themultispectral satellite images with thepublic GIS map datasets.
• This work won the fifth place in theBuilding Extraction Track of DeepGlobe- CVPR 2018 Satellite Challenge.

16×16×512
32×32×256

32×32×256
64×64×128
128×128×64
256×256×32

64×64×128

Convolution + Batch Normalization + Activation
Upsampling

Max-pooling

Concatenation

128×128×64
256×256×32
35 256×256×1

Case 3: More accurate urban land use map
Deep Convolutional
Neural Network

Rescaled images

Deep Convolutional
Neural Network

Satellite images
Sliced images

Deep Convolutional
Neural Network

Rescaled images

Satellite images
GIS Map images

Deep Convolutional
Neural Network
Sliced images

Augmentedimages

Semantic Segmentation

Probabilitymaps

Post-processing
Ensembling

Predictedbuildings36

Case 3: More accurate urban land use map
Results of the proposed method
Index

Vegas

Paris

Shanghai

Khartoum

TP27526

3097

11323

3495

Precision
Recall

0.94410.84370.84590.68250.74700.5396

0.63980.4694F1-score

0.89110.75550.6266

0.5415• Some examples of the building extraction results.

F1-scores obtained after each strategy
Method

Vegas

Paris

Shanghai

Khartoum

Baseline
Enhanced
Postprocess
Add-map

0.86110.87300.8866
0.89110.67740.71810.73830.75550.53420.54710.58970.62660.4544
0.49350.52100.5415• Our method improves the F1-scores by 3%-9% comparedwith the baseline, depending on the situation of each city.
• Combining the GIS Map datasets with satellite images
improves the results for all cities, even for places withfew building information on the map.

Vegas

ParisShanghai

Khartoum37

AI for Cityn Intelligent City Brain：p Detectionp Understanding
p Modelingp PlanningnLi, Weijia and Dong, Runmin and Fu, Haohuan and Wang, Jie and Yu, Le and Gong, Peng, "Integrating Google Earth imagery with Landsat
data to improve 30-m resolution land cover mapping", Remote Sensing of Environment, vol. 237, pp. 111563, 2020.

nZheng, Juepeng and Li, Weijia and Xia, Maocai and Dong, Runmin and Fu, Haohuan and Yuan, Shuai, "Large-Scale Oil Palm Tree Detection
from High-Resolution Remote Sensing Images Using Faster-RCNN", Proc. IEEE International Geoscience and Remote Sensing Symposium(IGARSS), pp. 1422-1425, 2019.

nDong, Runmin and Li, Weijia and Fu, Haohuan and Gan, Lin and Yu, Le and Zheng, Juepeng and Xia, Maocai, "Oil palm plantation mappingfrom high-resolution remote sensing images using deep learning", to appear in International Journal of Remote Sensing, pp. 1-25, 2019.

nXia, Maocai and Li, Weijia and Fu, Haohuan and Yu, Le and Dong, Runmin and Zheng, Juepeng, "Fast and robust detection of oil palm trees
using high-resolution remote sensing images", Proc. Automatic Target Recognition XXIX, vol. 10988, pp. 109880C, 2019.

nDong, Runmin and Li, Weijia and Fu, Haohuan and Xia, Maocai and Zheng, Juepeng and Yu, Le, "Semantic segmentation based large-scale
oil palm plantation detection using high-resolution satellite images", Proc. Automatic Target Recognition XXIX, vol. 10988, pp. 109880D,2019.

nLi, Weijia and He, Conghui and Fu, Haohuan and Zheng, Juepeng and Dong, Runmin and Xia, Maocai and Yu, Le and Luk, Wayne, "A RealTime Tree Crown Detection Approach for Large-Scale Remote Sensing Images on FPGAs", Remote Sensing, vol. 11, no. 9, pp. 1025, 2019.

nGong, Peng and Liu, Han and Zhang, Meinan and Li, Congcong and Wang, Jie and Huang, Huabing and Clinton, Nicholas and Ji, Luyan andLi, Wenyu and Bai, Yuqi and others, "Stable classification with limited sample: transferring a 30-m resolution sample set collected in 2015 tomapping 10-m resolution global land cover in 2017", Science Bulletin, vol. 64, no. 6, pp. 370-373, 2019.

nLi, Weijia and He, Conghui and Fang, Jiarui and Zheng, Juepeng and Fu, Haohuan and Yu, Le, "Semantic Segmentation-Based Building
Footprint Extraction Using Very High-Resolution Satellite Images and Multi-Source GIS Data", Remote Sensing, vol. 11, no. 4, pp. 403, 2019.

nLi, Weijia and Dong, Runmin and Fu, Haohuan and Yu, Le, “Large-scale oil palm tree detection from high-resolution satellite images using
two-stage convolutional neural networks”, Remote Sensing, vol. 11, no. 1, pp. 11, 2019.