Real Time Weather Recognition
based on Deep Learning
Collaborative work with Y. Yatsu, T. Yoshii, N. Kawai, J. Sakuma, N. Inoue, K. Shinoda (Tokyo Institute of Technology)

Overview

Astrophysics

In astronomy, the study of time-critical phenomena such as gravitational wave
events, supernovae and gamma-ray bursts is becoming more important. In order toperform optical follow-up observations of these events with ground-based telescope
immediately after the events occur, these observation systems need to be highlyautomated. However, observatories especially in Japan often suffer from clouds.
Visibility and sensitivity toward the particular field in the sky must be consideredwhen the follow-up observations are arranged. To improve observation efficiency,
we are developing a weather recognition system that is capable of automaticallyrecognizing the open sky, clouds and the moon in all-sky images by utilizing thedeep learning technology. We employ the images obtained by the all-sky camera
located at MITSuME Akeno observatory in Yamanashi, Japan for this development.

Weather Recognition SystemWeather recognition system for small
observatories- Ultra-low cost
- Recognition in real time
- High portability- Using the deep learning- Using commercially available devices

All-sky camera

Preparing Training DataBackground
Thick cloud

Clear skyThin cloud
Water drop

Sky imageLabels- Sampling 1,000 images- Defining 7 classes
Clear sky, Moon, Thin cloud,Thick cloud, Shadow, Water drop,
Background- Labeling these classes to the all
sampling images by eyes.

Akeno

Sky imageTokyo TechMITSuME Akeno 50cm telescope

Image

Algorithm & Training

Shadow

Pattern recognition

OutputDeep learning

GPUWeather recognition

Algorithm

TrainingThe algorithm proposed in “Condition Random
Fields as Recurrent Neural Networks(CRF-RNN)” is used. It consists of two stages:
Fully convolutional network (FCN) andCRF-RNN. A FCN predicts pixel-level labels
without considering structure. A CRF-RNN
stage performs CRF-based probabilistic
graphical modeling for structured prediction.

- Using a pre-trained model with MS COCO andPascal VOC- Fine tuning with the sky camera images- Resized from 2762x2762 pixels to 500x500
pixels to reduce the memory usage of a GPU- Randomly sampled 900 images for training andother 100 images for testing.
- Training the network for 130 epochs- Using the customized Caffe for CRF-RNN with
cuDNN running on NVIDIA GeForce GTX TITAN X

CRF-RNN trainablenetwork. Figure adaptedfrom Zheng et al.

Number of training images and test images that include
the classes.

Reference: S. Zheng, S. Jayasumana, B. Romera-Paredes, V. Vineet, Z. Su, D. Du,C.
Huang and P. Torr "Conditional Random Fields as Recurrent Neural Networks,”
The IEEE International Conference on Computer Vision (ICCV), 2015, pp. 1529-1537

Results & Evaluation
Fine nightwith the moonInput image

Recognizedby human

Predicted

Partially
cloudy nightwith the moon

Input image

Recognizedby human

Evaluation
Predicted

Intersection over Union (IoU) is used forthis evaluation. It is the standard metricused in the Pascal VOC challenge.

Prediction

Result Images

Partially
cloudy night
withoutthe moon

Dark night

True condition
It was “A”
It was not “A”

It may be
True positive False positive“A”
It may not be
False negative True negative“A”

Predictedd pd dnuu royo oulokwdrc clsogrrknk te ad ic ino eacaola
W Sh Th Th M CB

Very thinuniform cloud

Cloudy nightwith the moon

Background

TargetWater dropThin and thickclouds

Cloudy night
without
the moon

Shadow
Thick cloudThin cloud
Moon
Clear sky

Clear skyMoon

Thin cloudThick cloud

Shadow

Water drop

Background

1.0
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0.0

Visualization of the learnt label compatibility matrix.

http://www.gsic.titech.ac.jp/sc16