Package: ivtmetrics
ivtmetricsTheivtmetricslibrary provides a Python implementation of metrics
for benchmarking surgical action triplet detection and recognition.Features at a glanceThe following are available with ivtmetrics:1.Recognition
Evaluation: Provides AP metrics to measure the performance of a model
on action triplet recognition.2.Detection Evaluation: Supports
Intersection over Union distances measure of the triplet localization
with respect to the instruments.3.Flexible Analysis: (1) Supports
for switching between frame-wise to video-wise averaging of the AP. (2)
Supports disentangle prediction and obtained filtered performance for
the various components of the triplets as well as their association
performances at various levels.InstallationInstall via PyPiTo installivtmetricsusepippip install ivtmetricsInstall via Condaconda install -c nwoye ivtmetricsPython 3.5-3.9 and numpy and scikit-learn are required.MetricsThe metrics have been aligned with what is reported byCholecT50benchmark.ivtmetricscan be imported in the following way:importivtmetricsThe metrics implement bothrecognitionanddetectionevaluation.
The metrics internally implement a disentangle function to help filter
the triplet components as well as triplet different levels of
association.Recognition MetricsRecognition ivtmetricscan be used in the following ways:metric=ivtmetrics.Recognition(num_class)This takes an argumentnum_classwhich is default to100The following function are possible with theRecognitionclass:NameDescriptionupdate(targets, predictions)takes in a (batch of) vector
predictions and their
corresponding groundtruth. vector
size must matchnum_classin
the class initialization.video_end()Call to make the end of one video
sequence.reset()Reset current records. Useful
during training and can be called
at the begining of each epoch to
avoid overlapping epoch
performances.reset_global()Reset all records. Useful for
switching between
training/validation/testing or
can be called at the begining of
new experiment.compu
te_AP(component, ignore_null)Obtain the average precision on
the fly. This gives the AP only
on examples cases after the lastreset()call. Useful for
epoch performance during
training.compute_vid
eo_AP(component, ignore_null)(RECOMMENDED) compute video-wise
AP performance as used in
CholecT50 benchmarks.compute_glob
al_AP(component, ignore_null)compute frame-wise AP performance
for all seen samples.topK(k, component)Obtain top K performance on
action triplet recognition for
all seen examples. argskcan
be any int between 1-99. k =
[5,10,15,20] have been used in
benchmark papers.topClass(k, component)Obtain top K recognized classes
on action triplet recognition for
all seen examples. argskcan
be any int between 1-99. k = 10
have been used in benchmark
papers.args:argscomponentcan be any of the following (‘i’, ‘v’, ‘t’, ‘iv’,
‘it’,‘ivt’) to compute performance for (instrument, verb, target,
instrument-verb, instrument-target, instrument-verb-target)
respectively. default is ‘ivt’ for triplets.argsignore_null(optional, default=False): to ignore null
triplet classes in the evaluation. This option is enabled in
CholecTriplet2021 challenge.the output is adictwith keys(“AP”, “mAP”) for per-class and
mean AP respectively.Example usageimportivtmetricsrecognize=ivtmetrics.Recognition(num_class=100)network=MyModel(...)# your model here# trainingforepochinnumber-of-epochs:recognize.reset()forimages,labelsindataloader(...):# your data loaderpredictions=network(image)recognize.update(labels,predictions)results_i=recognize.compute_AP('i')print("instrument per class AP",results_i["AP"])print("instrument mean AP",results_i["mAP"])results_ivt=recognize.compute_AP('ivt')print("triplet mean AP",results_ivt["mAP"])# evaluationrecognize.reset_global()forvideoinvideos:forimages,labelsindataloader(video,..):# your data loaderpredictions=network(image)recognize.update(labels,predictions)recognize.video_end()results_i=recognize.compute_video_AP('i')print("instrument per class AP",results_i["AP"])print("instrument mean AP",results_i["mAP"])results_it=recognize.compute_video_AP('it')print("instrument-target mean AP",results_it["mAP"])results_ivt=recognize.compute_video_AP('ivt')print("triplet mean AP",results_ivt["mAP"])Anynanvalue in results is for classes with no occurrence in the
data sample.Detection MetricsDetection ivtmetricscan be used in the following ways:metric=ivtmetrics.Detection(num_class,num_tool,threshold=0.5)This takes an argumentnum_classwhich is default to100andnum_toolwhich is default to6The following function are possible with theDetectionclass:NameDescriptionupdate(targets, predictions, format)input: takes in a (batch of)
list/dict predictions and their
corresponding groundtruth. Each
frame prediction/groundtruth can
be either as alist of listor as alist of dict.
(more details below).video_end()Call to make the end of one video
sequence.reset()Reset current records. Useful
during training and can be called
at the begining of each epoch to
avoid overlapping epoch
performances.reset_global()Reset all records. Useful for
switching between
training/validation/testing or
can be called at the begining of
new experiment.compute_AP(component)Obtain the average precision on
the fly. This gives the AP only
on examples cases after the lastreset()call. Useful for
epoch performance during
training.compute_video_AP(component)(RECOMMENDED) compute video-wise
AP performance as used in
CholecT50 benchmarks.compute_global_AP(component)compute frame-wise AP performance
for all seen samples.args:list of list format: [[tripletID, toolID, toolProbs, x, y, w, h],
[tripletID, toolID, toolProbs, x, y, w, h], …], where:tripletID= triplet unique identitytoolID= instrument unique identitytoolProbs= instrument detection confidencex= bounding box x1 coordiantey= bounding box y1 coordinatew= width of the boxh= height of the boxThe [x,y,w,h] are scaled between 0..1list of dict format: [{“triplet”:tripletID, “instrument”:[toolID,
toolProbs, x, y, w, h]}, {“triplet”:tripletID, “instrument”:[toolID,
toolProbs, x, y, w, h]}, …].formatargs describes the input format with either of the values
(“list”, “dict”)componentcan be any of the following (‘i’, ‘v’, ‘t’, ‘iv’,
‘it’,‘ivt’) to compute performance for (instrument, verb, target,
instrument-verb, instrument-target, instrument-verb-target)
respectively, default is ‘ivt’ for triplets.the output is adictwith keys(“AP”, “mAP”, “Rec”, “mRec”, “Pre”,
“mPre”) for per-class AP, mean AP, per-class Recall, mean Recall,
per-class Precision and mean Precision respectively.Example usageimportivtmetricsdetect=ivtmetrics.Detection(num_class=100)network=MyModel(...)# your model here# trainingformat="list"forepochinnumberofepochs:forimages,labelsindataloader(...):# your data loaderpredictions=network(image)labels,predictions=formatYourLabels(labels,predictions)detect.update(labels,predictions,format=format)results_i=detect.compute_AP('i')print("instrument per class AP",results_i["AP"])print("instrument mean AP",results_i["mAP"])results_ivt=detect.compute_AP('ivt')print("triplet mean AP",results_ivt["mAP"])detect.reset()# evaluationformat="dict"forvideoinvideos:forimages,labelsindataloader(video,..):# your data loaderpredictions=network(image)labels,predictions=formatYourLabels(labels,predictions)detect.update(labels,predictions,format=format)detect.video_end()results_ivt=detect.compute_video_AP('ivt')print("triplet mean AP",results_ivt["mAP"])print("triplet mean recall",results_ivt["mRec"])print("triplet mean precision",results_ivt["mPre"])Anynanvalue in results is for classes with no occurrence in the
data sample.DisentangleAlthough, theDetection()andRecognition()classes uses theDisentangle()internally, this function can still be used independently for component filtering in the following ways:filter=ivtmetrics.Disentangle()Afterwards, any of the component’s predictions/labels can be filtered from the main triplet’s predictions/labels as follows:i_labels=filter.extract(inputs=ivt_labels,component="i")v_preds=filter.extract(inputs=ivt_preds,component="v")t_preds=filter.extract(inputs=ivt_preds,component="t")iv_labels=filter.extract(inputs=ivt_labels,component="iv")it_labels=filter.extract(inputs=ivt_labels,component="it")In this example, labels = (batch of) vector of groundtruth labels; preds = (batch of) vector of predicted probability scores.Dockercoming soon ..CitationIf you use this metrics in your project or research, please consider
citing the associated publication:@article{nwoye2022data,
   title={Data Splits and Metrics for Benchmarking Methods on Surgical Action Triplet Datasets},
   author={Nwoye, Chinedu Innocent and Padoy, Nicolas},
   journal={arXiv preprint arXiv:2204.05235},
   year={2022}
}ReferencesNwoye, C. I., Yu, T., Gonzalez, C., Seeliger, B., Mascagni, P.,
Mutter, D., … & Padoy, N. (2021). Rendezvous: Attention Mechanisms
for the Recognition of Surgical Action Triplets in Endoscopic Videos.
arXiv preprint arXiv:2109.03223.Nwoye, C. I., Gonzalez, C., Yu, T., Mascagni, P., Mutter, D.,
Marescaux, J., & Padoy, N. (2020, October). Recognition of
instrument-tissue interactions in endoscopic videos via action
triplets. In International Conference on Medical Image Computing and
Computer-Assisted Intervention (pp. 364-374). Springer, Cham.http://camma.u-strasbg.fr/datasetshttps://cholectriplet2021.grand-challenge.orghttps://cholectriplet2022.grand-challenge.orgLicenseBSD 2-Clause License

Copyright (c) 2022, Research Group CAMMA
All rights reserved.

Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are met:

1. Redistributions of source code must retain the above copyright notice, this
   list of conditions and the following disclaimer.

2. Redistributions in binary form must reproduce the above copyright notice,
   this list of conditions and the following disclaimer in the documentation
   and/or other materials provided with the distribution.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.```