Discriminative Learning of Deep Convolutional Feature Point Descriptors∗,1,5Edgar Simo-Serra∗,2,5, Eduard Trulls

3425, Luis Ferraz , Iasonas Kokkinos , Pascal Fua , Francesc Moreno-Noguer

1Waseda University, Tokyo, Japan 2 CVLab, École Polytechnique Fédérale de Lausanne, Switzerland 3 Catchoom Technologies, Barcelona, Spain
4CentraleSupelec and INRIA-Saclay, Chatenay-Malabry, France 5 Institut de Robòtica i Informàtica Industrial (CSIC-UPC), Barcelona, Spain (∗: First two authors contributed equally)

Model & Training• Learn compact, discriminative representations of image patcheswith Convolutional Neural Networks.
• Optimize for comparisons with the L2 distance, i.e. no metriclearning. Our descriptors work within existing pipelines.

Our model is a 3-Layer Convolutional Neural Network. For training we use a siamese architecture with weight sharing and SGD.
2Siamese network

Patches

38×8
5×5
128
4×4Tanh
4CNN

x1D(x1 )W
x2l(x1 , x2 )L2D(x2 )CNN

St.Dev.

1. We train a Siamese architecture with pairs of patches. We wantto bring matching pairs together and otherwise pull them apart.
2. Problem? Randomly sampled pairs are already easy to separate.
3. Solution: To train discriminative networks we use hard negativeand positive mining. This proves essential for performance.

0.8

0.8

0.7

0.7

0.7

0.6

0.6

0.6

0.4

SIFT (128f)
BGM (256b)
L-BGM (64f)
BinBoost-64 (64b)
BinBoost-128 (128b)
BinBoost-256 (256b)
VGG (80f)
CNN3, mined 8/8 (128f)

0.1
00

0.1

0.2

0.3

0.4

0.3
0.2
0.1

0.5

0.6

0.7

0.8

0.9

1

00

0.1

0.2

0.3

0.4

0.4
0.30.2
0.1

0.5

0.6

0.7

0.8

0.9

1000.1

0.2

0.3

Recall

0.4

0.5

0.6

0.7

0.8

0.9

1

RecallSIFT BGM L-BGM BinBoost-{64,128,256} VGG Ours
Test
(128f) (256b) (64f) (64b) (128b) (256b) (80f) (128f)

0.8
0.7
0.6

ND
YO
LY
All

0.5
0.4

SIFT (128f)
BGM (256b)
L-BGM (64f)
BinBoost-64 (64b)
BinBoost-128 (128b)
BinBoost-256 (256b)
VGG (80f)
CNN3, mined 8/8 (128f)

00.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

0.349
0.4250.226
0.370

0.487
0.495
0.268
0.440

0.495
0.517
0.355
0.508

0.267
0.283
0.202
0.291

0.451
0.457
0.346
0.469

0.549
0.533
0.410
0.550

0.663 0.667
0.709 0.545
0.558 0.608
0.693 0.756

1Recall

Generalization: Wide-Baseline MatchingThis penalizes corresponding pairs that are placed far apart, andnon-corresponding pairs that are less than C units apart.

Effect of mining
PR curve, training LY+YO, test ND

PR curve, training LY+ND, test YO

0.9

0.9

0.8

0.7

‘3’

‘5’

‘6’‘7’‘8’

0.4

Precision

Precision

Precision

0.5

0.4

Deformation Illumination

0.6
0.5
0.4

0.3
0.3

0.3

SIFT
CNN3, mined 1/2
CNN3, mined 2/2
CNN3, mined 4/4
CNN3, mined 8/8

0.2
0.1
00.2

00.1

0.2

0.3

0.4

SIFT
CNN3, mined 1/2
CNN3, mined 2/2
CNN3, mined 4/4
CNN3, mined 8/8

0.1

0.5

0.6

0.7

0.8

0.9

1

Recall

000.1

0.2

0.3

0.4

0.2
0.1

0.5

Recall

0.6

0.7

0.8

0.9

1

000.1

0.2

0.3

0.4

0.5

0.6

0.7

0.80.91Recallsp
sn PR AUC
128 1280.366
256 2560.374
512 5120.369
1024 1024 0.325

sp
sn rp rn Cost PR AUC
128 256 1 2 20% 0.558
256 256 2 2 35% 0.596
512 512 4 4 48% 0.703
1024 1024 8 8 67% 0.746

Table 1: (a) No mining.
Larger batches do not help.

Table 2: (b) Mining with rp = sp/128, rn = sn/128.
The mining cost is incurred during training only.

0.923
0.919
0.922
0.894
0.880
0.879
0.835
0.772

0.690
0.677
0.685
0.632
0.590
0.582
0.594
0.532

0.4560.424
0.439
0.400
0.372
0.365
0.363
0.308

0.218
0.197
0.228
0.174
0.182
0.166
0.172
0.138

0.088
0.058
0.058
0.067
0.058
0.064
0.032
0.053

Our models outperform the state-of-the-art on illumination changes
and non-rigid deformations [3] without re-training or fine-tuning.

0.7

0.5

Ours LY+YOOurs LY+ND
Ours YO+ND
VGG [4]YO
VGG [4]ND
VGG [4]LY
Daisy [6]–SIFT [2]–Generalization: Deformation and Illumination

0.6
0.6

‘4’Descriptor Training ‘3’ vs ‘4’ ‘3’ vs ‘5’ ‘3’ vs ‘6’ ‘3’ vs ‘7’ ‘3’ vs ‘8’

SIFT
CNN3, mined 1/2
CNN3, mined 2/2
CNN3, mined 4/4
CNN3, mined 8/8

0.90.7

Data from [5]. We match a set of points from view ‘3’ against ‘4’ to ‘8’(increasing baseline) and build PR curves, as before. No re-training.

PR curve, training YO+ND, test LY

10.8

0.8

This allows us to train discriminative models with a small numberof parameters (∼45k), which also alleviates overfitting concerns.

SIFT (128f)
BGM (256b)
L-BGM (64f)
BinBoost-64 (64b)
BinBoost-128 (128b)
BinBoost-256 (256b)
VGG (80f)
CNN3, mined 8/8 (128f)

0.5

0.901(e) Mined pairs

0.4

SIFT (128f)
BGM (256b)
L-BGM (64f)
BinBoost-64 (64b)
BinBoost-128 (128b)
BinBoost-256 (256b)
VGG (80f)
CNN3, mined 8/8 (128f)

PR curve, training "all"

10.1(a) Forward-propagate positives sp ≥ 128 and negatives sn ≥ 128.
(b) Pick the 128 with the largest loss (for each) and back-propagate.

(d) Random pairs

0.5

Recall0.2

Methodology: Train over two sets and test over third (leave-one-out),
with cross-validation. Metric: precision-recall (PR). ‘Needle in ahaystack’ setting: pick 10k unique points and generate one positive pair and 1k negative pairs for each, i.e. 10k positives vs. 10M
negatives. Results summarized by ‘Area Under the Curve’ (AUC).

We take samples from [1], for illustration. Corresponding patches are shown with same color:(a) Representation from t-SNE [8]. Distanceencodes similarity.
(b) Random sampling: similar (close) positives and different (distant) negatives.
(c) We mine the samples to obtain dissimilar
positives (+, long blue segments) and similar negatives (×, short red segments):(d) Random sampling results in easy pairs.
(e) Mined pairs with harder correspondences.

0.8

0.3

We minimize the hinge embedding loss. With 3D point indices p1, p2:
kD(x1) − D(x2)k2,
p 1 = p2l(x1, x2) =max(0, C − kD(x1) − D(x2)k2), p1 6= p2(c) “Hard” pairs: pos/neg

0.9

0.2

Key observation(a) 12 points/132 patches with t-SNE [8]0.90.5

PR curve, training YO+ND, test LY

10.90.3

Train on the MVS Dataset [1]. 64 × 64 grayscale patches from SFM:
Statue of Liberty (LY, top), NotreDame (ND, center), Yosemite (YO,bottom). ∼150k points and ∼450k patches each ⇒ 106 positive pairs
and 1012 negative pairs ⇒ Efficient exploration with mining.

PR curve, training LY+ND, test YO

1Precision

Input size
64 × 64 29 × 29
Filter size
7×7
6×6
Output channels
3264
Pooling & Norm.tion 2 × 23×3
Nonlinearity
TanhTanhStride23Mean(b) All pairs: pos/neg

PR curve, training LY+YO, test ND

1Precision• Drop-in replacement for SIFT: 128f, compare with the L2 norm.
• Consistent improvements over the state of the art.
• Trained in one dataset, but generalizes very well to scaling, rotation, deformation and illumination changes.
• Computational efficiency (on GPU: 0.76 ms; dense SIFT: 0.14 ms).
Code is available: https://github.com/etrulls/deepdesc-release1

We benchmark our models against SIFT, BinBoost [7], and VGG [4].
Better performance on 2/3 splits. Why? YO is very different from
LY/ND (e.g. mean/std). Training on all three sets: top performance.

Precision

Main features

Layer

Comparison with the state-of-the-art on MVS

Precision

Objective

Descriptor Training Def.
Ill. Def.+Ill.
Ours
LY+YO 76.568 88.434 75.933
Ours
LY+ND 75.702 87.521 75.606
Ours
YO+ND 76.731 88.898 76.591
VGG [4]YO
74.120 87.342 74.765
VGG [4]
ND 72.629 84.690 72.599
VGG [4]LY
72.602 84.848 72.565
DaLI [3]70.577 89.895 72.912
Daisy [6]67.373 75.402 66.197
SIFT [2]55.822 60.760 53.431

References[1] M. Brown, Gang Hua, and S. Winder. Discriminative learning of local image descriptors. PAMI, 2011.
[2] D. Lowe. Distinctive image features from scale-invariant keypoints. IJCV, 2004.
[3] E. Simo-Serra, C. Torras, and F. Moreno-Noguer. DaLI: Deformation and Light Invariant Descriptor. IJCV, 2015.
[4] K. Simonyan, A. Vedaldi, and A. Zisserman. Learning local feature descriptors using convex optimisation. PAMI,2014.
[5] C. Strecha, W. von Hansen, L. V. Gool, P. Fua, and U. Thoennessen. On benchmarking camera calibration andmulti-view stereo for high resolution imagery. In CVPR, 2008.
[6] E. Tola, V. Lepetit, and P. Fua. DAISY: An efficient dense descriptor applied to wide baseline stereo. PAMI, 2010.
[7] T. Trzcinski, M. Christoudias, P. Fua, and V. Lepetit. Boosting binary keypoint descriptors. In CVPR, 2013.
[8] L. van der Maaten and G. Hinton. Visualizing data using t-SNE. In JMLR, 2008.