WHAT’S NEW IN ENVI DEEPLEARNING 2.1
May 2023
Zach Norman
Bill Okubo

WHAT’S NEW IN ENVI DEEP LEARNING 2.1

1

AgendaIntroduction
Deep Learning

Machine Learning
Notables
Q+A

WHAT’S NEW IN ENVI DEEP LEARNING 2.1

2

Speakers Slide
Bill Okubo
Product Manager
bill.okubo@nv5.com

Zach Norman
Product Manager
zachary.norman@nv5.com
WHAT’S NEW IN ENVI DEEP LEARNING 2.1

3

Context: The ENVI Deep Learning Module
Applied deep learning for geospatial imagery in ENVI, theleading remote sensing and image analysis software

EvaluatePerformance

Without needing to program, the capabilities include:•Segmentation (i.e. cloud masking)•Object detection (i.e. cars or ships)•Linear feature extraction (i.e. roads)•Support for nearly any image format and data modality

CreateTraining
Data

Create Models

Deep learning workflow inENVI, built on TensorFlow andKeras
WHAT’S NEW IN ENVI DEEP LEARNING 2.1

Assess building damage
after hurricanes andtornadoes

Automated flood
detection using SAR

4

Context: ENVI Machine Learning
Applied machine learning for geospatial imagery in ENVI,
the leading remote sensing and image analysis software

Evaluate
Performance

CreateTraining
Data

Without needing to program, you have access to:•

Supervised classification (i.e. Random Forest)•Unsupervised classification (i.e. K-Means)•Anomaly detection (i.e. Isolation Forest)•Supports nearly any image format, data modality, or
number of bands

Create Models

Machine learning follows thesame workflow as deep learning,but takes a lot less time
WHAT’S NEW IN ENVI DEEP LEARNING 2.1

Get answers in minutes,
not hours using machinelearning

Use machine learning toprocess hyperspectral imagery
with any number of bands5

Machine Learning vs Deep Learning: Hardware and Access
MACHINE LEARNING• FREE and no additional license required, packaged with
“ENVI Deep Learning”• Available starting with ENVI 5.6.3• Uses the CPU• Recommended 16 GB RAM

DEEP LEARNING• Need to purchase the “ENVI Deep Learning” module to use• Latest version with be 2.0 and will have machine learning• Requires a GPU• Minimum 8 GB GPU RAM• Only NVIDIA GPUs are supported• Libraries optimized for Intel, but AMD also supported• Some algorithms have significant performance improvementswith IntelWHAT’S NEW IN ENVI DEEP LEARNING 2.1

6

DEEP LEARNING

WHAT’S NEW IN ENVI DEEP LEARNING 2.1

7

Deep Insights Using TensorBoard• TensorBoard is TensorFlow’svisualization engine for deep learningmodel performance• TensorBoard allows us to:– Visualize performance metrics (accuracy,precision recall, loss, etc.)– Display metrics from separate training
sessions in one chart– Create visualizations for other types of data,such as images, audio, or text

Screenshot of TensorBoard in ENVI Deep Learning, showing
some new metrics that are now reportedWHAT’S NEW IN ENVI DEEP LEARNING 2.1

8

The Old Way

The previous implementation of TensorBoard was hard to use and was not able to easily viewmetrics on the same graph

WHAT’S NEW IN ENVI DEEP LEARNING 2.1

9

The New Way

WHAT’S NEW IN ENVI DEEP LEARNING 2.1

10

Notable Changes• Visualize multiple metrics on one graph!
• X-axis represents number of samples that have been trained on– Allows comparisons between training sessions with different batch sizes on different GPUs

• As of ENVI Deep learning 2.0, we have had support for dark mode in TensorBoard• Each training session get’s three TensorBoard logs:– “batch” which covers each step of training– Updates each step of training– “training” which represents the performance of the model on data used directly for training– Once per epoch– “validation” which represents the performance of the model on data used for validation (i.e. evaluation)– Once per epoch

WHAT’S NEW IN ENVI DEEP LEARNING 2.1

11

Overall Metrics• Accuracy– How well the model re-creates its input– Higher is better• Precision– Measures the accuracy of positive
predictions– Higher is better• Recall– Measures the completeness of positive
predictions– Higher is better• Loss– The overall performance of the model– Lower is better

WHAT’S NEW IN ENVI DEEP LEARNING 2.1

A look at overall accuracy being reported in TensorBoard
which allows you to view accuracy during each step, foreach epoch, and for validation data in the same plot.

12

Types of Accuracy• Accuracy– Overall, traditional value

WHAT’S NEW IN ENVI DEEP LEARNING 2.1• Average class– Accuracy of each individualclass, averaged together• Average foreground class– Accuracy of each foregroundclassed, averaged together(i.e. features you areinterested in and have labeled)– Useful for rare or smallfeatures when backgroundoutnumbers foreground• Foreground– Accuracy for non-backgroundfeatures (i.e. features you areinterested in and have labeled)– Useful for rare or smallfeatures when backgroundoutnumbers foreground13Per Class Metrics• At the end of each epoch oftraining, we now report theaccuracy, precision, and recall ofeach class• You can view this for training andvalidation, but not for each batch• Depending on how rare yourfeatures are, and how muchtraining data you have, thesevalues may jump around a bit

WHAT’S NEW IN ENVI DEEP LEARNING 2.1

14

Multi-Class Class Metrics
Example of accuracy,
precision, and recall forforeground classes

Blue represents training andred is validation
Here’s the legend you’ll see
in TensorBoard

WHAT’S NEW IN ENVI DEEP LEARNING 2.1

15

Confusion Matrices

After each epoch, we generate confusion matrices for training and validation. For readability, therows are normalized from 0 to 1 (you don’t get raw class counts)
WHAT’S NEW IN ENVI DEEP LEARNING 2.1

16

Distributions and Histograms
Two new styles of metrics: distributions and histograms give insights into the range ofvalues for a metric in a single visual

Distributions show the range of values for agiven metric
WHAT’S NEW IN ENVI DEEP LEARNING 2.1

Histograms show the range of values for ametric along with the distribution of values
17Distributions and Histograms
Each plot shows the similar information, as shown by the red and light blue lines in thegraphs below, corresponding to extents of the values

WHAT’S NEW IN ENVI DEEP LEARNING 2.1

18

TensorBoard: Metrics vs Real-World PerformanceWith TensorBoard, youmight be surprised by whatgets reported for theaccuracy, precision, or recallof different classes
Zach’s pro tip is to always
process an image and havea look at the results yourself

Also keep in mind that we
can use additional postprocessing to clean up ourresults

WHAT’S NEW IN ENVI DEEP LEARNING 2.1

19

Reminder: Cleanup Tools
RASTER• Classification Aggregation– This task aggregates smaller class regions to a larger, adjacent region as part ofthe classification cleanup.

• Classification Clumping– This task performs a clumping method on a classification image. This operation
clumps adjacent similar classified areas using morphological operators.

• Classification Sieving– This task removes isolated classified pixels using blob grouping.

• Classification Smoothing– This task removes speckling noise from a classification image. It uses majority
analysis to change spurious pixels within a large single class to that class.

• Convert Classification to Vector Shapefile– Converts a classification image to a shapefile• Class Activation to Polyline ShapefileVECTOR• Filter Vector– Remove geometry from a vector based on area (pixels and normal units
such as meters• Smooth Vector– Reduce the number of vertices in a vector• Vector Record to Bounding Box– Convert polygons to a bounding box (extent-based or oriented with theshape of the features)• Vector Records to Centroid– Convert polygons to a centroid• Postprocess Classification Vector– Takes bounding boxes from object detection and allows you to do
additional filtering to cleanup results– Converts a linear mask of features to a polyline, where the polyline represents
the centerline (i.e polygon of roads converted to the centerline of roads)

WHAT’S NEW IN ENVI DEEP LEARNING 2.1

20

New Architecture: Pixel Segmentation• Creating a model for segmentation now
includes two architectures:– SegUNet++– SegUNet• Differences with the new model:– Better ability to detect features with smaller
training sessions– Generates less noise and fewer spurious (i.e.
standalone) pixels– About 10% faster– Slightly smaller network size

Architecture selection in the model
initialization dialog
WHAT’S NEW IN ENVI DEEP LEARNING 2.1

21

Model Comparison: Limited Training

WHAT’S NEW IN ENVI DEEP LEARNING 2.1

22

Tornado Damage 1

New Model(SegUNet++)
WHAT’S NEW IN ENVI DEEP LEARNING 2.1

Current Model(SegUNet)
23

Tornado Damage 2

New Model(SegUNet++)
WHAT’S NEW IN ENVI DEEP LEARNING 2.1

Current Model(SegUNet)24

Crosswalks 1

New Model(SegUNet++)
WHAT’S NEW IN ENVI DEEP LEARNING 2.1

Current Model(SegUNet)25

Crosswalks 2

New Model(SegUNet++)
WHAT’S NEW IN ENVI DEEP LEARNING 2.1

Current Model(SegUNet)
26

Crosswalks 3

New Model(SegUNet++)
WHAT’S NEW IN ENVI DEEP LEARNING 2.1

Current Model(SegUNet)
27

Crosswalks 4

New Model(SegUNet++)
WHAT’S NEW IN ENVI DEEP LEARNING 2.1

Current Model(SegUNet)28

Aircraft 1

New Model(SegUNet++)
WHAT’S NEW IN ENVI DEEP LEARNING 2.1

Current Model(SegUNet)29

Aircraft 2

New Model(SegUNet++)
WHAT’S NEW IN ENVI DEEP LEARNING 2.1

Current Model(SegUNet)30

Aircraft 3

New Model(SegUNet++)
WHAT’S NEW IN ENVI DEEP LEARNING 2.1

Current Model(SegUNet)
31

BREAKING CHANGE• The legacy “Single Class Model” can no longer be created in ENVI Deep Learning• This is an old concept and no longer applicable to the current state of the application• You can still process imagery using legacy single-class models

WHAT’S NEW IN ENVI DEEP LEARNING 2.1

32

Progress Reporting• Progress dialogs in ENVI when using ENVI DeepLearning were not always helpful or accurate• They would also show “Not Responding” while wewere waiting for message updates• TensorBoard was more useful to track progressbecause you would get more frequent updates

WHAT’S NEW IN ENVI DEEP LEARNING 2.1

33Progress Reporting: New User Experience• Now, all dialogs provide much
more insight into what is going on• Many more progress updates
provided for training to give you
progress through each epoch• Different types of deep learning
now present the same progressdialogs

Screenshots showing new dialogs and differences fromprevious versions

WHAT’S NEW IN ENVI DEEP LEARNING 2.1

34

MACHINE LEARNING

WHAT’S NEW IN ENVI DEEP LEARNING 2.1

35

Machine Learning: Normalization

We have improved the automatically generated
normalization statistics for machine learning to work
better with hyperspectral datasets.
However, if you know the range of your data (from0% to 100% surface reflectance) it is recommended
to use those values

WHAT’S NEW IN ENVI DEEP LEARNING 2.136

Updates: Random Forest and Extra Trees• New parameters for both:– Max features controls the number of features used to find thebest split– Custom max features lets you manually specify the numberof features used for the best fit• Random forest only:– OOB score tells ENVI Machine Learning to calculate andreturn the OOB (Out-of-Bag) score when it generates themodel

WHAT’S NEW IN ENVI DEEP LEARNING 2.137

Using OOB Score

Out-of-Bag (OOB) error rate plot from random forest classifiers
made with ENVI Machine Learning and the ENVI API in IDL

WHAT’S NEW IN ENVI DEEP LEARNING 2.1

38

NOTABLES

WHAT’S NEW IN ENVI DEEP LEARNING 2.1

39

ENVI Deep Learning: Updated Tutorials

Building damage tutorial updated with TensorBoard changes and includes new graphics
describing additional tips for the results and how to label data

WHAT’S NEW IN ENVI DEEP LEARNING 2.1

40

ENVI Deep Learning: Updated Tutorials

Legacy tutorial updated to use new architecture!

WHAT’S NEW IN ENVI DEEP LEARNING 2.1

41

Machine Learning: Algorithm Background

Want to know more about the algorithms included in ENVI Machine Learning? New descriptions of eachhave been added to the documentation!

WHAT’S NEW IN ENVI DEEP LEARNING 2.1

42

User Experience: Band Names

Images classified with ENVI Deep Learning or ENVI Machine Learning save the model type in the band
names, making it easy to compare results from different models without losing track of the source.
WHAT’S NEW IN ENVI DEEP LEARNING 2.1

43

Usability: New Tooltips

Like ENVI, the machine learning and deep learning
toolbox entries now have tooltips with basic
descriptions of each tool

WHAT’S NEW IN ENVI DEEP LEARNING 2.1

44

Questions and Discussion

Bill Okubo

Zach Norman

Product Manager

Product Manager

bill.okubo@nv5.com

zachary.norman@nv5.com

WHAT’S NEW IN ENVI DEEP LEARNING 2.1

NV5 Geospatial
www.NV5GeospatialSoftware.com
geospatialinfo@nv5.com
303-786-9900
45