#WWDC19

Understanding Images
in Vision Framework•

Brittany Weinert, Vision
Rohan Chandra, Vision
Sergey Kamensky, Vision© 2019 Apple Inc. All rights reserved. Redistribution or public display not permitted without written permission from Apple.

•Saliency•

Image Classification•

Image Similarity•

New detectors•

New revisions

•

Saliency

Brittany Weinert, Vision Team

What Catches Your Eye?

What Catches Your Eye?

Attention and Objectness Based Saliency

Attention and Objectness Based Saliency

Attention Based• Human Aspected• Trained on eye movements

Attention and Objectness Based Saliency

Attention Based• Human Aspected• Trained on eye movements

Objectness Based• Foreground Objects• Trained on object segmentation

Attention and Objectness Based Saliency

Attention and Objectness Based Saliency

Attention and Objectness Based Saliency

Attention and Objectness Based Saliency

Attention and Objectness Based Saliency

Attention and Objectness Based Saliency

Attention and Objectness Based Saliency

Attention and Objectness Based Saliency

Attention and Objectness Based Saliency

Contrast
Faces
Subjects
HorizonsLightAttention and Objectness Based Saliency

Attention and Objectness Based Saliency

Saliency Sample App

The Heatmap

The Heatmap

The Heatmap

The Heatmap

// Making a Request
let handler = VNImageRequestHandler(url: imageURL)
let request: VNImageBasedRequest = VNGenerateAttentionBasedSaliencyImageRequest()
request.revision = VNGenerateAttentionBasedSaliencyImageRequestRevision1
try? handler.perform([request])
guard let result = request.results?.firstlet observation = result as? VNSaliencyImageObservationelse { fatalError("missing result") }let pixelBuffer = observation.pixelBuffer// Making a Request
let handler = VNImageRequestHandler(url: imageURL)
let request: VNImageBasedRequest = VNGenerateAttentionBasedSaliencyImageRequest()
request.revision = VNGenerateAttentionBasedSaliencyImageRequestRevision1
try? handler.perform([request])
guard let result = request.results?.firstlet observation = result as? VNSaliencyImageObservationelse { fatalError("missing result") }let pixelBuffer = observation.pixelBuffer// Making a Request
let handler = VNImageRequestHandler(url: imageURL)
let request: VNImageBasedRequest = VNGenerateAttentionBasedSaliencyImageRequest()
request.revision = VNGenerateAttentionBasedSaliencyImageRequestRevision1
try? handler.perform([request])
guard let result = request.results?.firstlet observation = result as? VNSaliencyImageObservationelse { fatalError("missing result") }let pixelBuffer = observation.pixelBuffer// Making a Request
let handler = VNImageRequestHandler(url: imageURL)
let request: VNImageBasedRequest = VNGenerateAttentionBasedSaliencyImageRequest()
request.revision = VNGenerateAttentionBasedSaliencyImageRequestRevision1
try? handler.perform([request])
guard let result = request.results?.firstlet observation = result as? VNSaliencyImageObservationelse { fatalError("missing result") }let pixelBuffer = observation.pixelBuffer// Making a Request
let handler = VNImageRequestHandler(url: imageURL)
let request: VNImageBasedRequest = VNGenerateAttentionBasedSaliencyImageRequest()
request.revision = VNGenerateAttentionBasedSaliencyImageRequestRevision1
try? handler.perform([request])
guard let result = request.results?.firstlet observation = result as? VNSaliencyImageObservationelse { fatalError("missing result") }let pixelBuffer = observation.pixelBuffer// Making a Request
let handler = VNImageRequestHandler(url: imageURL)
let request: VNImageBasedRequest = VNGenerateAttentionBasedSaliencyImageRequest()
request.revision = VNGenerateAttentionBasedSaliencyImageRequestRevision1
try? handler.perform([request])
guard let result = request.results?.firstlet observation = result as? VNSaliencyImageObservationelse { fatalError("missing result") }let pixelBuffer = observation.pixelBuffer// Making a Request
let handler = VNImageRequestHandler(url: imageURL)
let request: VNImageBasedRequest = VNGenerateAttentionBasedSaliencyImageRequest()
request.revision = VNGenerateAttentionBasedSaliencyImageRequestRevision1
try? handler.perform([request])
guard let result = request.results?.firstlet observation = result as? VNSaliencyImageObservationelse { fatalError("missing result") }let pixelBuffer = observation.pixelBuffer// Making a Request
let handler = VNImageRequestHandler(url: imageURL)
let request: VNImageBasedRequest = VNGenerateAttentionBasedSaliencyImageRequest()
request.revision = VNGenerateAttentionBasedSaliencyImageRequestRevision1
try? handler.perform([request])
guard let result = request.results?.firstlet observation = result as? VNSaliencyImageObservationelse { fatalError("missing result") }let pixelBuffer = observation.pixelBuffer// Making a Request
let handler = VNImageRequestHandler(url: imageURL)
Attention BasedSaliencyImageRequest()
let request: VNImageBasedRequest = VNGenerateObjectness
Attention BasedSaliencyImageRequestRevision1
request.revision = VNGenerateObjectness
try? handler.perform([request])
guard let result = request.results?.firstlet observation = result as? VNSaliencyImageObservationelse { fatalError("missing result") }let pixelBuffer = observation.pixelBufferBounding Boxes(0,0)

func addSalientObjects(in observation: VNSaliencyImageObservation,
to path: CGMutablePath, transform:CGAffineTransform){
guard let objects = observation.salientObjects else { return }for object in objects {path.addRect(object.boundingBox, transform:transform)}}func addSalientObjects(in observation: VNSaliencyImageObservation,
to path: CGMutablePath, transform:CGAffineTransform){
guard let objects = observation.salientObjects else { return }for object in objects {path.addRect(object.boundingBox, transform:transform)}}func addSalientObjects(in observation: VNSaliencyImageObservation,
to path: CGMutablePath, transform:CGAffineTransform){
guard let objects = observation.salientObjects else { return }for object in objects {path.addRect(object.boundingBox, transform:transform)}}

Graphical Uses

Experiment with a different type of filter or photo transition

Graphical Uses

Experiment with a different type of filter or photo transition

Graphical Uses

Experiment with a different type of filter or photo transition

Working Together with Other Vision Algorithms

Use classification to recognize the salient objects

Working Together with Other Vision Algorithms

Use classification to recognize the salient objects

Working Together with Other Vision Algorithms

Use classification to recognize the salient objects

•Classification and Image Similarity

Rohan Chandra, Vision Team•Classification basics•

Taxonomy•

Interpreting observations•

Image Similarity

•Classification basics•

Taxonomy•

Interpreting observations•

Image Similarity

Classification in the Vision API

What is Classification
What objects arein the image?

kittenblanket

child

chair
table

bowl

textileDifficulties in Large Scale ClassificationDifficulties in Large Scale Classification

Annotated Data

Difficulties in Large Scale Classification

Annotated Data+

Compute PowerDifficulties in Large Scale Classification

Annotated Data+

Compute Power+Specialized ExpertiseClassification Powers Photos Search

Multi-label Versus Mono-label Classification

Multi-label Versus Mono-label Classification

Mono-label model
predicts “cat”
Multi-label model predicts“cat, adult, book”

•Classification basics•

Taxonomy•

Interpreting observations•

Image Similarity

•Classification basics•

Taxonomy•

Interpreting observations•

Image Similarity

Taxonomy

Taxonomy

Hierarchical structure,containing around 1000 classes

AnimalMammal

Dog
Cat

Pug
Corgi

BeagleTaxonomy

Hierarchical structure,containing around 1000 classes

AnimalMammalGrouping based on sharedsemantic meanings

Dog
Cat

Pug
Corgi

BeagleTaxonomy

Hierarchical structure,containing around 1000 classes

AnimalMammalGrouping based on sharedsemantic meanings

Dog
Cat

Defines relationships between
classes of increasing specificity

Pug
Corgi

BeagleTaxonomy

Hierarchical structure,containing around 1000 classes

AnimalMammalGrouping based on sharedsemantic meanings

Dog
Cat

Defines relationships between
classes of increasing specificity

Pug
Corgi

Beagle// List full taxonomy with
VNClassifiyImageRequest.knownClassifications(forRevision: VNClassifyImageRequestRevision1 )Taxonomy Construction

Taxonomy Construction

Include classes that are visually identifiableTaxonomy Construction

Include classes that are visually identifiable
Avoid• Abstract / controversial concepts

Taxonomy Construction

Include classes that are visually identifiable
Avoid• Abstract / controversial concepts• Proper nouns, adjectives, and basic shapes

Taxonomy Construction

Include classes that are visually identifiable
Avoid• Abstract / controversial concepts• Proper nouns, adjectives, and basic shapes• Occupations

Taxonomy Construction

Include classes that are visually identifiable
Avoid• Abstract / controversial concepts• Proper nouns, adjectives, and basic shapes• Occupations

let handler = VNImageRequestHandler(url: imageUrl)
let request = VNClassifyImageRequest()
try? handler.perform([request])
let observations = request.results as? [VNClassificationObservation]

let handler = VNImageRequestHandler(url: imageUrl)
let request = VNClassifyImageRequest()
try? handler.perform([request])
let observations = request.results as? [VNClassificationObservation]

let handler = VNImageRequestHandler(url: imageUrl)
let request = VNClassifyImageRequest()
try? handler.perform([request])
let observations = request.results as? [VNClassificationObservation]

Observations output:(<Label>, <Confidence>)[
let handler = VNImageRequestHandler(url: imageUrl)
let request = VNClassifyImageRequest()
try? handler.perform([request])
let observations = request.results as? [VNClassificationObservation](animal, 0.848),(cat, 0.848),(mammal, 0.848),(clothing, 0.676),(beanie, 0.675),(hat, 0.675),(people, 0.616),(adult, 0.616),(snow, 0.445),(jacket, 0.214),(outdoor, 0.063),(leash, 0.057),(cord, 0.057),…

•Classification basics•

Taxonomy•

Interpreting observations•

Image Similarity

•Classification basics•

Taxonomy•

Interpreting observations•

Image Similarity

Interpreting Observation, Terms

Interpreting Observation, Terms

Confidence > Threshold =>Predicted motorcycle
in image

Interpreting Observation, Terms

Confidence > Threshold =>Predicted motorcycle
in image

Confidence < Threshold =>Predicted no
motorcycle in image

Precision and Recall

Precision and Recall
RecallPercentage of Target Class retrieved from entire libraryObservations output:(<Label>, <Confidence>)[(animal, 0.848),
let handler = VNImageRequestHandler(url: imageUrl)(cat, 0.848),

let request = VNClassifyImageRequest()(mammal, 0.848),

try? handler.perform([request])(clothing, 0.676),

let observations = request.results as? [VNClassificationObservation](beanie, 0.675),

let searchObservations = observations?.filter { $0.hasMinimumPrecision(0.0,(hat, 0.675),

forRecall: 0.7) }(people, 0.616),(adult, 0.616),(snow, 0.445),(jacket, 0.214),(outdoor, 0.063),(leash, 0.057),(cord, 0.057),…
]

[(animal, 0.848),
let handler = VNImageRequestHandler(url: imageUrl)(cat, 0.848),

let request = VNClassifyImageRequest()(mammal, 0.848),

try? handler.perform([request])(clothing, 0.676),

let observations = request.results as? [VNClassificationObservation](beanie, 0.675),

let searchObservations = observations?.filter { $0.hasMinimumPrecision(0.0,(hat, 0.675),

forRecall: 0.7) }(people, 0.616),(adult, 0.616),(snow, 0.445),(jacket, 0.214),]

Precision and Recall

Precision and Recall
Precision
Percentage of returned results that are target classObservations output:(<Label>, <Confidence>)[(animal, 0.848),
let handler = VNImageRequestHandler(url: imageUrl)(cat, 0.848),

let request = VNClassifyImageRequest()(mammal, 0.848),

try? handler.perform([request])(clothing, 0.676),

let observations = request.results as? [VNClassificationObservation](beanie, 0.675),

let searchObservations = observations?.filter { $0.hasMinimumRecall(0.0,(hat, 0.675),

forPrecision: 0.7) }(people, 0.616),(adult, 0.616),(snow, 0.445),(jacket, 0.214),(outdoor, 0.063),(leash, 0.057),(cord, 0.057),…
]

let handler = VNImageRequestHandler(url: imageUrl)[

let request = VNClassifyImageRequest()(animal, 0.848),

try? handler.perform([request])(cat, 0.848),

let observations = request.results as? [VNClassificationObservation](mammal, 0.848),

let searchObservations = observations?.filter { $0.hasMinimumRecall(0.0,(clothing, 0.676),

forPrecision: 0.7) }(beanie, 0.675),(hat, 0.675),(people, 0.616),(adult, 0.616),(snow, 0.445),]

The PR Curve

The PR Curve

Operating point where
Recall = 0.7
Precision = 0.74

The PR Curve
Operating point where
Recall = 0.5
Precision = 0.92
Operating point where
Recall = 0.7
Precision = 0.74

Operating point where
Recall = 0.8
Precision = 0.26

Filtering the PR Curvelet observations = request.results as? [VNClassificationObservation]
let searchObservations = observations?.filter { $0.hasMinimumPrecision(0.4, forRecall: 0.5) }Filtering the PR Curvelet observations = request.results as? [VNClassificationObservation]
let searchObservations = observations?.filter { $0.hasMinimumPrecision(0.4, forRecall: 0.5) }precision > 0.4

precision > 0.4

precision > 0.4

Filtering the PR Curvelet observations = request.results as? [VNClassificationObservation]
let searchObservations = observations?.filter { $0.hasMinimumPrecision(0.4, forRecall: 0.5) }precision > 0.4

precision > 0.4

recall = 0.5

precision > 0.4

recall = 0.5

recall = 0.5

Filtering the PR Curvelet observations = request.results as? [VNClassificationObservation]
let searchObservations = observations?.filter { $0.hasMinimumPrecision(0.4, forRecall: 0.5) }SelectedThresholdprecision > 0.4

precision > 0.4

SelectedThresholdprecision > 0.4

No viable
Thresholdrecall = 0.5

recall = 0.5

recall = 0.5

Classification: Summary

Returned Observation
contains labels and an
associated confidence[

Choice of threshold isapplication specific(feline, 0.82),

Can be determined bydesired precision and recall(outdoor, 0.37),(wood_natural, 0.90),(animal, 0.82),(mammal, 0.82),(cat, 0.82),(snow, 0.51),(land, 0.37),…
]

•Classification basics•

Taxonomy•

Interpreting observations•

Image Similarity

•Classification basics•

Taxonomy•

Interpreting observations•

Image Similarity

Image Similarity

How can we describe an image?

Image Similarity

Describe an image using pixels?

Input Image

Image Similarity

Describe an image using pixels?

Input Image

Fails to retrieve

Image Similarity

Describe an image using words?

Input Image

Image Similarity

Describe an image using words?

Input Image
Semantically Similar
Word search
Kitten,Bowl

Image Similarity

Describe an image using words?

Input Image
Semantically Similar
Word search
Kitten,Bowl

Image Similarity

Image Similarity

Descriptor should describes image content, not just appearanceImage Similarity

Descriptor should describes image content, not just appearance
Classification network creates representations of images

Image Similarity

Descriptor should describes image content, not just appearance
Classification network creates representations of images
FeaturePrint — vector image descriptor similar to a word vector

Image Similarity

Descriptor should describes image content, not just appearance
Classification network creates representations of images
FeaturePrint — vector image descriptor similar to a word vector

19.28

23.85

Pair-wise distance

25.54

Image Similarity

Image Similarity

Image Similarity

Image Similarity

Image Similarity

Image Similarity

Image Similarity

•

Demo

•

Image Similarity game// Generate featureprints for copies and compute distances from original featureprint
for index in contestantImageURLs.indices {let contestantImageURL = contestantImageURLs[index]
if let contestantFPO = featureprintObservationForImage(atURL: contestantImageURL) {do {var distance = Float(0)try contestantFPO.computeDistance(&distance, to: originalFPO)
ranking.append((contestantIndex: index, featureprintDistance: distance))} catch {print("Error computing distance between featureprints.")}}}

// Generate featureprints for copies and compute distances from original featureprint
for index in contestantImageURLs.indices {let contestantImageURL = contestantImageURLs[index]
if let contestantFPO = featureprintObservationForImage(atURL: contestantImageURL) {do {var distance = Float(0)try contestantFPO.computeDistance(&distance, to: originalFPO)
ranking.append((contestantIndex: index, featureprintDistance: distance))} catch {print("Error computing distance between featureprints.")}}}

// Generate featureprints for copies and compute distances from original featureprint
for index in contestantImageURLs.indices {let contestantImageURL = contestantImageURLs[index]
if let contestantFPO = featureprintObservationForImage(atURL: contestantImageURL) {do {var distance = Float(0)try contestantFPO.computeDistance(&distance, to: originalFPO)
ranking.append((contestantIndex: index, featureprintDistance: distance))} catch {print("Error computing distance between featureprints.")}}}

Face Technology, New Detectors,
Tracking, and CoreML•

Sergey Kamensky, Vision Team•

Face Technology

Face Landmarks

Face Landmarks
NewFace Landmarks
New

65pt

76pt

Face Landmarks
New

65pt
Single Confidence Score

76pt
Confidence Score per point

Face Landmarks

Pupils Detection

NewFace Landmarks

ExampleFace Landmarks

Examplelet request = VNDetectFaceLandmarksRequest()
let requestHandler = VNImageRequestHandler(url: imageURL)
try requestHandler.perform([request])
let faceObservation = request.results!.first! as! VNFaceObservation
let landmarks2D = faceObservation.landmarks!

Face Landmarks

Examplelet request = VNDetectFaceLandmarksRequest()
let requestHandler = VNImageRequestHandler(url: imageURL)
try requestHandler.perform([request])
let faceObservation = request.results!.first! as! VNFaceObservation
let landmarks2D = faceObservation.landmarks!

Face Landmarks

Examplelet request = VNDetectFaceLandmarksRequest()
let requestHandler = VNImageRequestHandler(url: imageURL)
try requestHandler.perform([request])
let faceObservation = request.results!.first! as! VNFaceObservation
let landmarks2D = faceObservation.landmarks!

Face Landmarks

Examplelet request = VNDetectFaceLandmarksRequest()
let requestHandler = VNImageRequestHandler(url: imageURL)
try requestHandler.perform([request])
let faceObservation = request.results!.first! as! VNFaceObservation
let landmarks2D = faceObservation.landmarks!

Face Landmarks

Examplelet request = VNDetectFaceLandmarksRequest()
let requestHandler = VNImageRequestHandler(url: imageURL)
try requestHandler.perform([request])
let faceObservation = request.results!.first! as! VNFaceObservation
let landmarks2D = faceObservation.landmarks!

Face Landmarks

Examplelet request = VNDetectFaceLandmarksRequest()
let requestHandler = VNImageRequestHandler(url: imageURL)
try requestHandler.perform([request])
let faceObservation = request.results!.first! as! VNFaceObservation
let landmarks2D = faceObservation.landmarks!

VNDetectedObjectObservation
boundingBox

VNFaceObservation…
landmarks…

Face Landmarks

Examplelet request = VNDetectFaceLandmarksRequest()
let requestHandler = VNImageRequestHandler(url: imageURL)
try requestHandler.perform([request])
let faceObservation = request.results!.first! as! VNFaceObservation
let landmarks2D = faceObservation.landmarks!

VNDetectedObjectObservation

VNFaceLandmarks

boundingBox

confidence

VNFaceObservation

VNFaceLandmarks2D…
landmarks…

Face Contour
Eyes
Eyebrows
Nose
Lips
Pupils…

Face Landmarks

Examplelet request = VNDetectFaceLandmarksRequest()
let requestHandler = VNImageRequestHandler(url: imageURL)
try requestHandler.perform([request])
let faceObservation = request.results!.first! as! VNFaceObservation
let landmarks2D = faceObservation.landmarks!

VNDetectedObjectObservation

VNFaceLandmarks

boundingBox

confidence

VNFaceLandmarkRegion

VNFaceLandmarks2D

pointCount

Face Contour
Eyes
Eyebrows
Nose
Lips
Pupils…

VNFaceLandmarkRegion2D

VNFaceObservation…
landmarks…

normalizedPoints
imageSize
precisionEstimatesPerPointFace Landmarks

Examplelet request = VNDetectFaceLandmarksRequest()
let requestHandler = VNImageRequestHandler(url: imageURL)
try requestHandler.perform([request])
let faceObservation = request.results!.first! as! VNFaceObservation
let landmarks2D = faceObservation.landmarks!

VNDetectedObjectObservation

VNFaceLandmarks

boundingBox

confidence

VNFaceLandmarkRegion

VNFaceLandmarks2D

pointCount

Face Contour
Eyes
Eyebrows
Nose
Lips
Pupils…

VNFaceLandmarkRegion2D

VNFaceObservation…
landmarks…

normalizedPoints
imageSize
precisionEstimatesPerPointFace Landmarks

Revisioning example — default versus explicit
let request = VNDetectFaceLandmarksRequest()
let requestHandler = VNImageRequestHandler(url: imageURL)
try requestHandler.perform([request])
let faceObservation = request.results!.first! as! VNFaceObservation
let lm2D = faceObservation.landmarks!

Face Landmarks

Revisioning example — default versus explicit
let request = VNDetectFaceLandmarksRequest()
let requestHandler = VNImageRequestHandler(url: imageURL)
try requestHandler.perform([request])
let faceObservation = request.results!.first! as! VNFaceObservation
let lm2D = faceObservation.landmarks!

Face Landmarks

Revisioning example — default versus explicit
let request = VNDetectFaceLandmarksRequest()
request.revision = VNDetectFaceLandmarksRequestRevision2// request.constellation = VNRequestFaceLandmarksConstellation65Points
let requestHandler = VNImageRequestHandler(url: imageURL)
try requestHandler.perform([request])
let faceObservation = request.results!.first! as! VNFaceObservation
let lm2D = faceObservation.landmarks!

2018 SDKs

Face Landmarks

Revisioning example — default versus explicit
let request = VNDetectFaceLandmarksRequest()
request.revision = VNDetectFaceLandmarksRequestRevision3
request.constellation = VNRequestFaceLandmarksConstellation76Points
let requestHandler = VNImageRequestHandler(url: imageURL)
try requestHandler.perform([request])
let faceObservation = request.results!.first! as! VNFaceObservation
let lm2D = faceObservation.landmarks!

2019 SDKs

Face Landmarks

Revisioning example — default versus explicit
let request = VNDetectFaceLandmarksRequest()
request.revision = VNDetectFaceLandmarksRequestRevision3
request.constellation = VNRequestFaceLandmarksConstellation76Points
let requestHandler = VNImageRequestHandler(url: imageURL)
try requestHandler.perform([request])
let faceObservation = request.results!.first! as! VNFaceObservation
let lm2D = faceObservation.landmarks!

2019 SDKs

Face Capture Quality

Face Capture Quality

Face Capture Quality is a holistic measure that considers:lighting, blur, occlusion, expression, pose, etc.

Face Capture Quality

0.561

0.150

Face Capture Quality is a holistic measure that considers:lighting, blur, occlusion, expression, pose, etc.

Face Capture Quality

Examplelet request = VNDetectFaceCaptureQualityRequest()
let requestHandler = VNImageRequestHandler(url: imageURL)
try requestHandler.perform([request])
let faceObservation = request.results!.first! as! VNFaceObservation
let faceCaptureQuality = faceObservation.faceCaptureQuality

0.625

Face Capture Quality

Examplelet request = VNDetectFaceCaptureQualityRequest()
let requestHandler = VNImageRequestHandler(url: imageURL)
try requestHandler.perform([request])
let faceObservation = request.results!.first! as! VNFaceObservation
let faceCaptureQuality = faceObservation.faceCaptureQuality

0.625

Face Capture Quality

Examplelet request = VNDetectFaceCaptureQualityRequest()
let requestHandler = VNImageRequestHandler(url: imageURL)
try requestHandler.perform([request])
let faceObservation = request.results!.first! as! VNFaceObservation
let faceCaptureQuality = faceObservation.faceCaptureQuality

0.625

VNDetectedObjectObservation
boundingBox

VNFaceObservation…
faceCaptureQuality
…

float

Face Capture Quality

Face Capture Quality

0.409

0.513

0.430

0.302

0.349

0.333

Face Capture Quality

0.409

0.513

0.430

0.302

0.349

0.333

0.302

0.333

0.349

0.409

0.430

0.513

Face Capture Quality

0.409

0.513

0.430

0.302

0.349

0.333

0.302

0.333

0.349

0.409

0.430

0.513

Face Capture Quality

…
0.302

…
0.513

0.418

0.625

Face Capture Quality

…
0.302

0.0

…
0.513

0.418

0.625

Face Capture Quality

Face Capture Quality

…
0.302

0.0

…
0.513

0.418

0.625

Face Capture Quality

Face Capture Quality should not be compared against a thresholdFace Capture Quality

…
0.302

0.0

…
0.513

0.418

0.520

0.625

Face Capture Quality

Face Capture Quality should not be compared against a thresholdFace Capture Quality

…
0.302

0.0

…
0.513

0.418

0.520

0.625

Face Capture Quality

Face Capture Quality should not be compared against a threshold
Face Capture Quality is a comparative measure of the same subject

•New Detectors

New Detectors - Human Detector

New Detectors - Cat and Dog Detectors

Dog
Cat

Dog

Cat

Dog

New Detectors

Examples// Human Detector
let request = VNDetectHumanRectanglesRequest()
let requestHandler = VNImageRequestHandler(url: imageURL)
try requestHandler.perform([request])
let human = request.results!.first! as! VNDetectedObjectObservation

uuid

1234

boundingBox [0.2, 0.3, 0.4, 0.8]// Animal Detector
let request = VNDetectAnimalRectanglesRequest()
let requestHandler = VNImageRequestHandler(url: imageURL)
try requestHandler.perform([request])
let dog = request.results!.first! as! VNRecognizedObjectObservation

uuid

5678

boundingBox [0.2, 0.4, 0.5, 0.8]
labels{“dog”}

New Detectors

Examples// Human Detector
let request = VNDetectHumanRectanglesRequest()
let requestHandler = VNImageRequestHandler(url: imageURL)
try requestHandler.perform([request])
let human = request.results!.first! as! VNDetectedObjectObservation

uuid

1234

boundingBox [0.2, 0.3, 0.4, 0.8]// Animal Detector
let request = VNDetectAnimalRectanglesRequest()
let requestHandler = VNImageRequestHandler(url: imageURL)
try requestHandler.perform([request])
let dog = request.results!.first! as! VNRecognizedObjectObservation

uuid

5678

boundingBox [0.2, 0.4, 0.5, 0.8]
labels{“dog”}

New Detectors

Examples// Human Detector
let request = VNDetectHumanRectanglesRequest()
let requestHandler = VNImageRequestHandler(url: imageURL)
try requestHandler.perform([request])
let human = request.results!.first! as! VNDetectedObjectObservation

uuid

1234

boundingBox [0.2, 0.3, 0.4, 0.8]// Animal Detector
let request = VNDetectAnimalRectanglesRequest()
let requestHandler = VNImageRequestHandler(url: imageURL)
try requestHandler.perform([request])
let dog = request.results!.first! as! VNRecognizedObjectObservation

uuid

5678

boundingBox [0.2, 0.4, 0.5, 0.8]
labels{“dog”}

•

Tracking Enhancements

Tracking with Vision

Tracking with VisionLess expansion into the backgroundTracking with VisionLess expansion into the background
Better handling of occlusions

Tracking with VisionLess expansion into the background
Better handling of occlusions
Machine Learning basedTracking with VisionLess expansion into the background
Better handling of occlusions
Machine Learning based
Runs on CPU, GPU, and A12 Bionic with low power consumption

New Object Tracker

New Object Tracker

Tracking with Vision

Examplelet requestHandler = VNSequenceRequestHandler()var inputObservation = VNDetectedObjectObservation(boundingBox: objectBoundingBox)
for _ in 1...5 {let frame = frameFeeder.nextFrame()
let request = VNTrackObjectRequest(detectedObjectObservation: inputObservation)
try requestHandler.perform([request], on: frame)
let observation = request.results!.first! as! VNDetectedObjectObservation// Process new object location returned in observation.boundingBox
inputObservation = observation}

Tracking with Vision

Examplelet requestHandler = VNSequenceRequestHandler()var inputObservation = VNDetectedObjectObservation(boundingBox: objectBoundingBox)
for _ in 1...5 {let frame = frameFeeder.nextFrame()
let request = VNTrackObjectRequest(detectedObjectObservation: inputObservation)
try requestHandler.perform([request], on: frame)
let observation = request.results!.first! as! VNDetectedObjectObservation// Process new object location returned in observation.boundingBox
inputObservation = observation}

Tracking with Vision

Examplelet requestHandler = VNSequenceRequestHandler()var inputObservation = VNDetectedObjectObservation(boundingBox: objectBoundingBox)
for _ in 1...5 {let frame = frameFeeder.nextFrame()
let request = VNTrackObjectRequest(detectedObjectObservation: inputObservation)
try requestHandler.perform([request], on: frame)
let observation = request.results!.first! as! VNDetectedObjectObservation// Process new object location returned in observation.boundingBox
inputObservation = observation}

Tracking with Vision

Examplelet requestHandler = VNSequenceRequestHandler()var inputObservation = VNDetectedObjectObservation(boundingBox: objectBoundingBox)
for _ in 1...5 {let frame = frameFeeder.nextFrame()
let request = VNTrackObjectRequest(detectedObjectObservation: inputObservation)
request.revision = VNTrackObjectRequestRevision2
try requestHandler.perform([request], on: frame)
let observation = request.results!.first! as! VNDetectedObjectObservation// Process new object location returned in observation.boundingBox
inputObservation = observation}

•

Vision and Core ML

Vision and CoreML Integration Enhancements

Vision now works with Core ML models that have single input of image type

Vision and CoreML Integration Enhancements

Vision now works with Core ML models that have single input of image type
Vision converts ‘Inputs’ image to Core ML required input size and color scheme

Vision and CoreML Integration Enhancements

Vision now works with Core ML models that have single input of image type
Vision converts ‘Inputs’ image to Core ML required input size and color scheme
Vision wraps ‘Outputs’ into appropriate Observation types

Vision and CoreML Integration Enhancements

Vision and CoreML Integration Enhancements

Vision can now work with Core ML models that have one or more ‘Inputs’• Including multi-image inputs

Vision and CoreML Integration Enhancements

Vision can now work with Core ML models that have one or more ‘Inputs’• Including multi-image inputs

Vision will use name-mapping of ‘Output’ names to Observations

Vision and CoreML Integration Enhancements

Vision can now work with Core ML models that have one or more ‘Inputs’• Including multi-image inputs

Vision will use name-mapping of ‘Output’ names to Observations

Vision and CoreML Integration Flow - New API
let mlModel = try MLModel(contentsOf: modelURL)
let visionModel = try VNCoreMLModel(for: mlModel)
visionModel.inputImageFeatureName = "imageContent"
visionModel.featureProvider =["imageStyle" : MLFeatureValue.featureValueWithPixelBuffer(pixelBuffer)"mixRatio" : MLFeatureValue.featureValueWithDouble(0.3)]
let request = VNCoreMLRequest(model: visionModel)
let requestHandler = VNImageRequestHandler(url: imageContentURL) // "imageContent"
try requestHandler.perform([request])
let results = request.results!
for case let resultingImage as VNPixelBufferObservation in results {if resultingImage.featureName == "imageResult" { /* process results */ }}More Information
developer.apple.com/wwdc19/222

Text Recognition in Vision Framework

Thursday, 4:00

Machine Learning LabsWWDC 2019