Package: patchnetvlad
Patch-NetVLAD: Multi-Scale Fusion of Locally-Global Descriptors for Place RecognitionThis repository contains code for the CVPR2021 paper "Patch-NetVLAD: Multi-Scale Fusion of Locally-Global Descriptors for Place Recognition"The article can be found onarXivand theofficial proceedings.License + attribution/citationWhen using code within this repository, please refer the followingpaperin your publications:@inproceedings{hausler2021patchnetvlad,
  title={Patch-NetVLAD: Multi-Scale Fusion of Locally-Global Descriptors for Place Recognition},
  author={Hausler, Stephen and Garg, Sourav and Xu, Ming and Milford, Michael and Fischer, Tobias},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  pages={14141--14152},
  year={2021}
}The code is licensed under theMIT License.InstallationWe recommend using conda (or better: mamba) to install all dependencies. If you have not yet installed conda/mamba, please download and installmambaforge.# On Linux:condacreate-npatchnetvladpythonnumpypytorch-gputorchvisionnatsorttqdmopencvpillowscikit-learnfaissmatplotlib-base-cconda-forge# On MacOS (x86 Intel processor):condacreate-npatchnetvladpythonnumpypytorchtorchvisionnatsorttqdmopencvpillowscikit-learnfaissmatplotlib-base-cconda-forge# On MacOS (ARM M1/M2 processor):condacreate-npatchnetvladpythonnumpypytorchtorchvisionnatsorttqdmopencvpillowscikit-learnfaissmatplotlib-base-cconda-forge-ctobiasrobotics# On Windows:condacreate-npatchnetvladpythonnumpynatsorttqdmopencvpillowscikit-learnfaissmatplotlib-base-cconda-forge
condainstallpytorchtorchvisiontorchaudiopytorch-cuda=11.7-cpytorch-cnvidia

condaactivatepatchnetvladWe provide several pre-trained models and configuration files. The pre-trained models will be downloaded automatically into thepretrained_modelsthe first time feature extraction is performed.Alternatively, you can manually download the pre-trained models into a folder of your choice; click to expand if you want to do so.We recommend downloading the models into thepretrained_modelsfolder (which is setup in the config files within theconfigsdirectory):# Note: the pre-trained models will be downloaded automatically the first time feature extraction is performed# the steps below are optional!# You can use the download script which automatically downloads the models:python./download_models.py# Manual download:cdpretrained_models
wget-Omapillary_WPCA128.pth.tarhttps://cloudstor.aarnet.edu.au/plus/s/vvr0jizjti0z2LR/download
wget-Omapillary_WPCA512.pth.tarhttps://cloudstor.aarnet.edu.au/plus/s/DFxbGgFwh1y1wAz/download
wget-Omapillary_WPCA4096.pth.tarhttps://cloudstor.aarnet.edu.au/plus/s/ZgW7DMEpeS47ELI/download
wget-Opittsburgh_WPCA128.pth.tarhttps://cloudstor.aarnet.edu.au/plus/s/2ORvaCckitjz4Sd/download
wget-Opittsburgh_WPCA512.pth.tarhttps://cloudstor.aarnet.edu.au/plus/s/WKl45MoboSyB4SH/download
wget-Opittsburgh_WPCA4096.pth.tarhttps://cloudstor.aarnet.edu.au/plus/s/1aoTGbFjsekeKlB/downloadIf you want to use the shortcutspatchnetvlad-match-two,patchnetvlad-feature-matchandpatchnetvlad-feature-extract, you also need to run (which also lets you use Patch-NetVLAD in a modular way):pip3install--no-deps-e.Quick startFeature extractionReplaceperformance.iniwithspeed.iniorstorage.iniif you want, and adapt the dataset paths - examples are given for the Pittsburgh30k dataset (simply replacepitts30kwithtokyo247ornordlandfor these datasets).pythonfeature_extract.py\--config_pathpatchnetvlad/configs/performance.ini\--dataset_file_path=pitts30k_imageNames_index.txt\--dataset_root_dir=/path/to/your/pitts/dataset\--output_features_dirpatchnetvlad/output_features/pitts30k_indexRepeat for the query images by replacing_indexwith_query. Note that you have to adaptdataset_root_dir.Feature matching (dataset)pythonfeature_match.py\--config_pathpatchnetvlad/configs/performance.ini\--dataset_root_dir=/path/to/your/pitts/dataset\--query_file_path=pitts30k_imageNames_query.txt\--index_file_path=pitts30k_imageNames_index.txt\--query_input_features_dirpatchnetvlad/output_features/pitts30k_query\--index_input_features_dirpatchnetvlad/output_features/pitts30k_index\--ground_truth_pathpatchnetvlad/dataset_gt_files/pitts30k_test.npz\--result_save_folderpatchnetvlad/results/pitts30kNote that providingground_truth_pathis optional.This will create three output files in the folder specified byresult_save_folder:recalls.txtwith a plain text output (only ifground_truth_pathis specified)NetVLAD_predictions.txtwith top 100 reference images for each query images obtained using "vanilla" NetVLAD inKapture formatPatchNetVLAD_predictions.txtwith top 100 reference images from above re-ranked by Patch-NetVLAD, again inKapture formatFeature matching (two files)pythonmatch_two.py\--config_pathpatchnetvlad/configs/performance.ini\--first_im_path=patchnetvlad/example_images/tokyo_query.jpg\--second_im_path=patchnetvlad/example_images/tokyo_db.pngWe provide thematch_two.pyscript which computes the Patch-NetVLAD features for two given images and then determines the local feature matching between these images. While we provide example images, any image pair can be used.The script will print a score value as an output, where a larger score indicates more similar images and a lower score means dissimilar images. The function also outputs a matching figure, showing the patch correspondances (after RANSAC) between the two images. The figure is saved asresults/patchMatchings.png.Trainingpythontrain.py\--config_pathpatchnetvlad/configs/train.ini\--cache_path=/path/to/your/desired/cache/folder\--save_path=/path/to/your/desired/checkpoint/save/folder\--dataset_root_dir=/path/to/your/mapillary/datasetTo begin, request, download and unzip the Mapillary Street-level Sequences dataset (https://github.com/mapillary/mapillary_sls).
The provided script will train a new network from scratch, to resume training add --resume_path and set to a full path, filename and extension to an existing checkpoint file. Note to resume our provided models, first remove the WPCA layers.After training a model, PCA can be added using add_pca.py.pythonadd_pca.py\--config_pathpatchnetvlad/configs/train.ini\--resume_path=full/path/with/extension/to/your/saved/checkpoint\--dataset_root_dir=/path/to/your/mapillary/datasetThis will add an additional checkpoint file to the same folder as resume_path, except including a WPCA layer.FAQHow to Create New Ground Truth FilesWe provide three ready-to-go ground truth files in the dataset_gt_files folder, however, for evaluation on other datasets you will need to create your own .npz ground truth data files.
Each .npz stores three variables:utmQ(a numpy array of floats),utmDb(a numpy array of floats) andposDistThr(a scalar numpy float).Each successive element withinutmQandutmDbneeds to correspond to the corresponding row of the image list file.posDistThris the ground truth tolerance value (typically in meters).The following mock example details the steps required to create a new ground truth file:Collect GPS data for your query and database traverses and convert to utm format. Ensure the data is sampled at the same rate as your images.Select your own choice of posDistThr value.Save these variables using Numpy, such as this line of code:np.savez('dataset_gt_files/my_dataset.npz', utmQ=my_utmQ, utmDb=my_utmDb, posDistThr=my_posDistThr)AcknowledgementsWe would like to thank Gustavo Carneiro, Niko Suenderhauf and Mark Zolotas for their valuable comments in preparing this paper. This work received funding from the Australian Government, via grant AUSMURIB000001 associated with ONR MURI grant N00014-19-1-2571. The authors acknowledge continued support from the Queensland University of Technology (QUT) through the Centre for Robotics.Related worksPlease check outthis collectionof related works on place recognition.