Package: brown-phototour-revisited
brown_phototour_revisitedThe package for local patch descriptors evaluation, which takes into account image indexes  and second nearest neighbor ratio (SNN) filtering strategy. So, it evaluates descriptors in a similar way, how they are used in practice. It is in agreement withIMC benchmark, unlike the original protocol. The benchmark is not "test benchmark" by amy means. Rather it is intended to be used as validation/development set for local patch descriptor learning and/or crafting.Installpip install brown_phototour_revisitedHow to useThere is a single function, which does everything for you:full_evaluation. The original Brown benchmark consider evaluation, similar to cross-validation: train descriptor on one subset, evaluate on two others, repeat for all, so 6 evaluations are required. For the handcrafted descriptors, or those, that are trained on 3rd party datasets, only 3 evaluations are necessary.  We are following it here as well.However, if you need to run some tests separately, or reuse some functions -- we will cover the usage below.
In the following example we will show how to usefull_evaluationto evaluate SIFT descriptor as implemented in kornia.# !pip install korniaimport torch
import kornia
from IPython.display import clear_output
from brown_phototour_revisited.benchmarking import *
patch_size = 65 

model = kornia.feature.SIFTDescriptor(patch_size, rootsift=True).eval()

descs_out_dir = 'data/descriptors'
download_dataset_to = 'data/dataset'
results_dir = 'data/mAP'

results_dict = {}
results_dict['Kornia RootSIFT'] = full_evaluation(model,
                                'Kornia RootSIFT',
                                path_to_save_dataset = download_dataset_to,
                                path_to_save_descriptors = descs_out_dir,
                                path_to_save_mAP = results_dir,
                                patch_size = patch_size, 
                                device = torch.device('cuda:0'), 
                           distance='euclidean',
                           backend='pytorch-cuda')
clear_output()
print_results_table(results_dict)------------------------------------------------------------------------------
Mean Average Precision wrt Lowe SNN ratio criterion on UBC Phototour Revisited
------------------------------------------------------------------------------
trained on       liberty notredame  liberty yosemite  notredame yosemite
tested  on           yosemite           notredame            liberty
------------------------------------------------------------------------------
Kornia RootSIFT        56.70              47.71               48.09 
------------------------------------------------------------------------------Precomputed benchmark resultsWe have pre-computed some results for you. The implementation is in the following notebooks in theexamplesdir:Deep descriptorsNon-deep descriptorsThe final tables are the following:------------------------------------------------------------------------------
Mean Average Precision wrt Lowe SNN ratio criterion on UBC Phototour Revisited
------------------------------------------------------------------------------
trained on       liberty notredame  liberty yosemite  notredame yosemite
tested  on           yosemite           notredame            liberty
------------------------------------------------------------------------------
Kornia RootSIFT 32px   58.24              49.07               49.65 
HardNet 32px       70.64  70.31        61.93  59.56        63.06  61.64
SOSNet 32px        70.03  70.19        62.09  59.68        63.16  61.65
TFeat 32px         65.45  65.77        54.99  54.69        56.55  56.24
SoftMargin 32px    69.29  69.20        61.82  58.61        62.37  60.63
HardNetPS 32px         55.56              49.70               49.12 
R2D2_center_grayscal   61.47              53.18               54.98 
R2D2_MeanCenter_gray   62.73              54.10               56.17 
------------------------------------------------------------------------------

------------------------------------------------------------------------------
Mean Average Precision wrt Lowe SNN ratio criterion on UBC Phototour Revisited
------------------------------------------------------------------------------
trained on       liberty notredame  liberty yosemite  notredame yosemite
tested  on           yosemite           notredame            liberty
------------------------------------------------------------------------------
Kornia SIFT 32px       58.47              47.76               48.70 
OpenCV_SIFT 32px       53.16              45.93               46.00 
Kornia RootSIFT 32px   58.24              49.07               49.65 
OpenCV_RootSIFT 32px   53.50              47.16               47.37 
OpenCV_LATCH 65px  -----  -----        -----  37.26        -----  39.08
OpenCV_LUCID 32px      20.37              23.08               27.24 
skimage_BRIEF 65px     52.68              44.82               46.56 
Kornia RootSIFTPCA 3 60.73  60.64        50.80  50.24        52.46  52.02
MKD-concat-lw-32 32p 72.27  71.95        60.88  58.78        60.68  59.10
------------------------------------------------------------------------------Disclaimer 1: don't trust this table fullyI haven't (yet!) checked if all the deep descriptors models, trained on Brown, were trained with flip-rotation 90 degrees augmentation. In the code below I assume that they were, however, it might not be true -- and the comparison might not be completely fair. I will do my best to check it, but if you know that I have used wrong weights - please open an issue. Thank you.Disclaimer 2: it is not "benchmark".The intended usage of the package is not to test and report the numbers in the paper. Instead think about is as cross-validation tool, helping the development. Thus, one CAN tune hyperparameters based on the benchmark results  instead of doing so onHPatches. After you have finished tuning, please, evaluate your local descriptors on some downstream task likeIMC image matching benchmarkorvisual localization.If you found any mistake, please open an issueDetailed examples of usageThere are 3 main modules of the package:dataset,extractionandbenchmarking.To run the benchmark manually one needs two things:extract the descriptors with eitherextract_pytorchinput_descriptorsorextract_numpyinput_descriptorsget the mean average precision (mAP) withevaluate_mAP_snn_basedHere we will show how to evaluate several descriptors: Pytorch-based HardNet, OpenCV SIFT, skimage BRIEF.The code below will download the HardNet, trained on Liberty dataset, download the Notredame subset and extracts the local patch descriptors into the dict. Note, that we should not evaluate descriptor on the same subset, as it was trained on.import torch
import kornia

from brown_phototour_revisited.dataset import *
from brown_phototour_revisited.extraction import *
from brown_phototour_revisited.benchmarking import *

model = kornia.feature.HardNet(True).eval()

descs_out_dir = 'data/descriptors'
download_dataset_to = 'data/dataset'
patch_size = 32 # HardNet expects 32x32 patches

desc_dict = extract_pytorchinput_descriptors(model,
                                'HardNet+Liberty',
                                subset= 'notredame', 
                                path_to_save_dataset = download_dataset_to,
                                path_to_save_descriptors = descs_out_dir,
                                patch_size = patch_size, 
                                device = torch.device('cuda:0'))# Found cached data data/dataset/notredame.pt
data/descriptors/HardNet+Liberty_32px_notredame.npy already exists, loadingprint (desc_dict.keys())dict_keys(['descriptors', 'labels', 'img_idxs'])Functionextract_pytorchinput_descriptorsexpectstorch.nn.Module, which takes(B, 1, patch_size, patch_size)torch.Tensorinput and outputs(B, desc_dim)torch.Tensor.Now we will calculate mAP.mAP = evaluate_mAP_snn_based(desc_dict['descriptors'],
                             desc_dict['labels'], 
                             desc_dict['img_idxs'],
                             path_to_save_mAP = 'data/mAP/HardNet+Liberty_notredame.npy',
                            backend='pytorch-cuda')
print (f'HardNetLib mAP on Notredame = {mAP:.5f}')Found saved results data/mAP/HardNet+Liberty_notredame.npy, loading
HardNetLib mAP on Notredame = 0.61901Now we will evaluate OpenCV SIFT descriptor.
Functionextract_numpyinput_descriptorsexpects function or object, which takes (patch_size, patch_size) input and outputs (desc_dim) np.array.As OpenCV doesn't provide such function, we will create it ourselves, or rather take already implemented fromHPatches benchmark repoimport cv2
import numpy as np
patch_size = 65

# https://github.com/hpatches/hpatches-benchmark/blob/master/python/extract_opencv_sift.py#L43
def get_center_kp(PS=65.):
    c = PS/2.0
    center_kp = cv2.KeyPoint()
    center_kp.pt = (c,c)
    center_kp.size = PS/5.303
    return center_kp


sift = cv2.SIFT_create()
center_kp = get_center_kp(patch_size)

def extract_opencv_sift(patch):
    #Convert back to UINT8 and provide aux keypoint in the center of the patch
    return sift.compute((255*patch).astype(np.uint8),[center_kp])[1][0].reshape(128)

descs_out_dir = 'data/descriptors'
download_dataset_to = 'data/dataset'


desc_dict_sift = extract_numpyinput_descriptors(extract_opencv_sift,
                                'OpenCV_SIFT',
                                subset= 'notredame', 
                                path_to_save_dataset = download_dataset_to,
                                path_to_save_descriptors = descs_out_dir,
                                patch_size = patch_size)# Found cached data data/dataset/notredame.pt
data/descriptors/OpenCV_SIFT_65px_notredame.npy already exists, loadingmAP_SIFT = evaluate_mAP_snn_based(desc_dict_sift['descriptors'],
                             desc_dict_sift['labels'], 
                             desc_dict_sift['img_idxs'],
                            path_to_save_mAP = 'data/mAP/OpenCV_SIFT65_notredame.npy',
                            backend='pytorch-cuda')
print (f'OpenCV SIFT PS = {patch_size}, mAP on Notredame = {mAP_SIFT:.5f}')Found saved results data/mAP/OpenCV_SIFT65_notredame.npy, loading
OpenCV SIFT PS = 65, mAP on Notredame = 0.45530Now, let's try some binary descriptor, like BRIEF. Evaluation so far supports two metrics:"euclidean"and"hamming".Functionextract_numpyinput_descriptorsexpects function or object, which takes(patch_size, patch_size)input and outputs(desc_dim)np.array.As skimage doesn't provide exactly such function, we will create it ourselves by placing "detected" keypoint in the center of the patch.import numpy as np
from skimage.feature import BRIEF
patch_size = 65
BR = BRIEF(patch_size = patch_size)
def extract_skimage_BRIEF(patch):
    BR.extract(patch.astype(np.float64), np.array([patch_size/2.0, patch_size/2.0]).reshape(1,2))
    return BR.descriptors.astype(np.float32)

desc_dict_brief = extract_numpyinput_descriptors(extract_skimage_BRIEF,
                                'skimage_BRIEF',
                                subset= 'notredame', 
                                path_to_save_dataset = download_dataset_to,
                                path_to_save_descriptors = descs_out_dir,
                                patch_size = patch_size)# Found cached data data/dataset/notredame.pt
data/descriptors/skimage_BRIEF_65px_notredame.npy already exists, loadingThat's will take a while.mAP_BRIEF = evaluate_mAP_snn_based(desc_dict_brief['descriptors'].astype(np.bool),
                             desc_dict_brief['labels'], 
                             desc_dict_brief['img_idxs'],
                             path_to_save_mAP = 'data/mAP/skimageBRIEF_notredame.npy',
                             backend='numpy',
                             distance='hamming')
print (f'skimage BRIEF PS = {patch_size}, mAP on Notredame = {mAP_BRIEF:.5f}')Found saved results data/mAP/skimageBRIEF_notredame.npy, loading
skimage BRIEF PS = 65, mAP on Notredame = 0.44817Loading cached resultsYou can also directly load already calculated results from cache without creating a model by using functionload_cached_resultsfrom brown_phototour_revisited.benchmarking import load_cached_results
desc_name = 'HardNet'
patch_size = 32
desc_dict =  load_cached_results(desc_name,
                    learned_on = ['liberty', 'notredame', 'yosemite'],
                    path_to_save_dataset = download_dataset_to,
                    path_to_save_descriptors = descs_out_dir,
                    path_to_save_mAP = results_dir,
                    patch_size = patch_size)

results_dict[f'{desc_name} {patch_size}px'] = desc_dict
clear_output()
print_results_table(results_dict)------------------------------------------------------------------------------
Mean Average Precision wrt Lowe SNN ratio criterion on UBC Phototour Revisited
------------------------------------------------------------------------------
trained on       liberty notredame  liberty yosemite  notredame yosemite
tested  on           yosemite           notredame            liberty
------------------------------------------------------------------------------
Kornia RootSIFT        56.70              47.71               48.09 
HardNet 32px       70.64  70.31        61.93  59.56        63.06  61.64
------------------------------------------------------------------------------If you use the benchmark, please cite it:@misc{BrownRevisited2020,
  title={UBC PhotoTour Revisied},
  author={Mishkin, Dmytro},
  year={2020},
  url = {https://github.com/ducha-aiki/brown_phototour_revisited}
}