—DECEMBER 02, 2019

Perception Challenges in Industrial Applications

Remus Boca, ABB Future Labs

ASTM International Committee on 3D Imaging Systems (E57), NIST Workshop 
—

December 3, 2019 Slide 2

Perception Challenges in Industrial Applications

Bio

Bin Picking

2008

https://www.youtube.com/watch?v=09LzufOnbX0
—

Agenda

December 3, 2019 Slide 3

Perception Challenges in Industrial Applications

As the world moves towards autonomy, the sensing and perception are 

becoming (critical) important. 

Industrial applications have their own challenges as they operate in possible 

harsh environments, they require continuous and robust operation, need to 

accommodate unstructured environments, acknowledge a wide range of states 

and unexpected events. 

This talk presents perception needs and challenges across many industries such 

as ports, mining, transformer inspection and robotics.
—

Ports

December 3, 2019 Slide 4

Perception Challenges in Industrial Applications

Ship to shore cranes (IN / OUT)

Stacking cranes operating in the shipyard

Straddle carriers

Port gates (IN / OUT)

https://new.abb.com/ports
—

December 3, 2019 Slide 5

Perception Challenges in Industrial Applications

Ports

Multiple vision modalities for different tasks

– remote control

– pick / drop container

– safety

Outdoor operation

– Enclosure is needed

– environment conditions : day / night; sunny, 

cloudy, rain, snow, wind; humidity 

Working range 

– from 1m to 30m (stacking crane)

– from 1m to 85 m (ship to shore)

Accuracy medium to low (cm)

Access to sensors is required

Objects of interest wear and tear

– container & container stacks

– chassis

– ships / bays
—

December 3, 2019 Slide 6

Perception Challenges in Industrial Applications

Mining

https://new.abb.com/news/detail/11252/abb-ability-conveyor-roller-inspection-services
—

December 3, 2019 Slide 7

Perception Challenges in Industrial Applications

Mining

Several thousands idlers to be 

inspected

Idlers moving at high speed

Idlers placement

Measured regularly and quickly

Environment conditions:

Vibration

Shock

Dust

Heat

Moisture

Rain / cloudy / snow / 

sunny

Low cost

Robustness

Geometric, texture and & 

physical properties sensing

Shape

Texture

Temperature

Audio

Low power budget

https://new.abb.com/news/detail/11252/abb-ability-conveyor-roller-inspection-services
—

December 3, 2019 Slide 8

Transformer

inspection

Perception Challenges in Industrial Applications

https://new.abb.com/products/transformers/service/advanced-services/txplore
—

December 3, 2019 Slide 9

Perception Challenges in Industrial Applications

Transformer inspection

Operating environment

It can be contaminated

Decontaminate after every 

use

Wide range of temperatures

Resolution for quality images

Field of view / surround

Short distance

Lighting 

More than one sensor

Many types of defects

Appearance

Geometric

Wireless communication

Sensors are critical to operate 

while submersed (expensive to 

recover the device)

redundant cameras

https://new.abb.com/products/transformers/service/advanced-services/txplore
—

December 3, 2019 Slide 10

Perception Challenges in Industrial Applications

Transformer Inspection

ABB Ability TXplore

https://www.youtube.com/watch?v=OclzoEc8plE
—

December 3, 2019 Slide 11

Perception Challenges in Industrial Applications

Wire assembly

A380

Total wire count ~100 000

Total wire length ~470Km

Total weight of wires 5700 Kg

~30 % weight to fix cables to the structure
—

Accuracy sub-mm

High resolution

High frame rate

Small and large field of view

Small footprint

Short range

Quality components (lens, 

global shutter)

Multiple sensors working on 

the same field of view

Low cost

Lifecycle as long as a robot

Easy diagnostic

Industrial communication 

protocols

Easy customization

December 3, 2019 Slide 12

Perception Challenges in Industrial Applications

Wire assembly

Luxury car

Total wire count ~1500

Total wire length ~1 Mile 

Total weight of wires 150 #
—

December 3, 2019 Slide 13

Perception Challenges in Industrial Applications

Ports

Multiple modalities for different tasks

– remote control

– pick / drop container

– safety

Outdoor operation

– enclosure

– environment conditions : day / night; 

sunny, cloudy, rain, snow, wind; 

humidity 

Working range 

– from 1m to 30m (stacking crane)

– from 1m to 85 m (ship to shore)

Accuracy medium to low (cm)

Access to sensors

Objects of interest wear and tear

– container & container stacks

– chassis

– ships / bays

Mining

Several thousands idlers to be 

inspected

Idlers moving at high speed

Idlers placement

Measured regularly and quickly

Outdoor & environment 

conditions:

Vibration

Shock

Dust

Heat

Moisture

Rain / cloudy / snow / 

sunny

Low cost

Robustness

Geometric, texture and & 

physical properties sensing

Shape

Texture

Temperature

Audio

Low power budget

Transformer inspection

Operating environment

It can be contaminated

Decontaminate after 

every use

Wide range of 

temperatures

Resolution for quality images

Field of view / surround

Short distance

Lighting 

More than one sensors

Many types of defects

Appearance

Geometric

Wireless communication

Sensors are critical to 

operate while submersed 

(expensive to recover the 

device)

redundant 

cameras

Wire assembly

Accuracy sub-mm

High resolution

High frame rate

Small and large field of view

Small footprint

Short range

Quality components (lens, 

global shutter)

Multiple sensors working on 

the same field of view

Low cost

Lifecycle as long as a robot

Easy diagnostic

Industrial communication 

protocols

Easy customization