Package: uotod
Unbalanced Optimal Transport: A Unified Framework for Object DetectionPresentationPaperSupplementaryDocumentationH. De Plaen, P.-F. De Plaen, J. A. K. Suykens, M. Proesmans, T. Tuytelaars, and L. Van Gool, “Unbalanced Optimal Transport: A Unified Framework for Object Detection,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Jun. 2023, pp. 3198–3207.This work has be presented at CVPR 2023 in Vancouver, Canada. The paper and additional resources can be found on thefollowing website. The paper is in open access and can also be found on theCVF websiteas well as onIEEE Xplore.AbstractTL;DR: We introduce a much more versatile new class of matching strategies unifying many existing ones, as well as being well suited for GPUs.During training, supervised object detection tries to correctly match the predicted bounding boxes and associated classification scores to the ground truth. This is essential to determine which predictions are to be pushed towards which solutions, or to be discarded. Popular matching strategies include matching to the closest ground truth box (mostly used in combination with anchors), or matching via the Hungarian algorithm (mostly used in anchor-free methods). Each of these strategies comes with its own properties, underlying losses, and heuristics. We show how Unbalanced Optimal Transport unifies these different approaches and opens a whole continuum of methods in between. This allows for a finer selection of the desired properties. Experimentally, we show that training an object detection model with Unbalanced Optimal Transport is able to reach the state-of-the-art both in terms of Average Precision and Average Recall as well as to provide a faster initial convergence. The approach is well suited for GPU implementation, which proves to be an advantage for large-scale models.InstallPyPIUsing PyPI, it suffices to runpip install uotod. Just rerun this command to update the package to its newest version.Build From SourceYou can also download it directly from the GitHub repository, then build and install it.gitclone--recursivehttps://github.com/hdeplaen/uotodcduotod
python3-mpipinstall-rrequirements.txt
python3-msetupbuild
python3-mpipinstallCompiled AccelerationThe package isavailable on all dsitributionsand runs well. However, only the combinations marked with a green ✅ can
take advantage of the compiled version of Sinkhorn's algorithm directly from PyPI. On a not support combination, you may always build it
from the source to also have access to Sinkhorn's compiled version of the algorithm. Nevertheless, the PyTorch implementation
ofSinkhorn's algorithm is always available(used by default), this only refers to an additional compiled version.OSLinuxMacOSWindowsPython 3.8✅✅☑️Python 3.9✅✅☑️Python 3.10✅✅☑️Python 3.11✅✅☑️Python 3.12✅☑️☑️✅: Python implementation + compiled acceleration,both directly from PyPI☑️: Python implementationdirectly from PyPI(+ possible compiled acceleration if building from source)ExamplesOT matching with GIoU loss:fromuotod.matchimportBalancedSinkhornfromuotod.lossimportGIoULossot=BalancedSinkhorn(loc_match_module=GIoULoss(reduction="none"),background_cost=0.,)Hungarian matching (bipartite) with GIoU loss:fromuotod.matchimportHungarianfromuotod.lossimportGIoULosshungarian=Hungarian(loc_match_module=GIoULoss(reduction="none"),background_cost=0.,)Loss from SSD solved with Unbalanced Optimal Transport:fromtorch.nnimportL1Loss,CrossEntropyLossfromuotod.matchimportUnbalancedSinkhornfromuotod.lossimportDetectionLoss,IoULossmatching_method=UnbalancedSinkhorn(cls_match_module=None,# No classification costloc_match_module=IoULoss(reduction="none"),background_cost=0.5,# Threshold for matching to backgroundis_anchor_based=True,# Use anchor-based matchingreg_target=1e-3,# Relax the constraint that each ground truth is matched to exactly one prediction)loss=DetectionLoss(matching_method=matching_method,cls_loss_module=CrossEntropyLoss(reduction="none"),loc_loss_module=L1Loss(reduction="none"),)preds=...targets=...anchors=...loss_value=loss(preds,targets,anchors)Loss from DETR solved with Optimal Transport (num_classes=3):importtorchfromtorch.nnimportL1Loss,CrossEntropyLossfromuotod.matchimportBalancedSinkhornfromuotod.lossimportDetectionLossfromuotod.lossimportMultipleObjectiveLoss,GIoULoss,NegativeProbLossmatching_method=BalancedSinkhorn(cls_match_module=NegativeProbLoss(reduction="none"),loc_match_module=MultipleObjectiveLoss(losses=[GIoULoss(reduction="none"),L1Loss(reduction="none")],weights=[1.,5.],),background_cost=0.,# Does not influence the matching when using balanced OT)loss=DetectionLoss(matching_method=matching_method,cls_loss_module=CrossEntropyLoss(reduction="none",weight=torch.tensor([0.1,1.,1.])# down-weight the loss for the no-object class),loc_loss_module=MultipleObjectiveLoss(losses=[GIoULoss(reduction="none"),L1Loss(reduction="none")],weights=[1.,5.],),)preds=...targets=...loss_value=loss(preds,targets)Color Boxes DatasetCitationIf you make any use of our work, please refer to us as:@InProceedings{De_Plaen_2023_CVPR,author={De Plaen, Henri and De Plaen, Pierre-Fran\c{c}ois and Suykens, Johan A. K. and Proesmans, Marc and Tuytelaars, Tinne and Van Gool, Luc},title={Unbalanced Optimal Transport: A Unified Framework for Object Detection},booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},month={June},year={2023},pages={3198-3207}}AcknowledgementsEU: The research leading to these results has received funding from the European Research Council under the European Union’s Horizon 2020 research and innovation program / ERC Advanced Grant E-DUALITY (787960). This paper reflects only the authors’ views and the Union is not liable for any use that may be made of the contained information. Research Council KUL: Optimization frameworks for deep kernel machines C14/18/068. Flemish Government: FWO: projects: GOA4917N (Deep Restricted Kernel Machines: Methods and Foundations), PhD/Postdoc grant; This research received funding from the Flemish Government (AI Research Program). All the authors are also affiliated to Leuven.AI - KU Leuven institute for AI, B-3000, Leuven, Belgium.