Package: lvt-eval
lvt-evalInstallationpython>=3.8, (windows: need c++ env,https://airesources.oss-cn-hangzhou.aliyuncs.com/jkl/%E8%BE%B9%E7%BC%98%E5%8D%A1/VisualStudioSetup.exe)pip install pybind11  -ihttps://pypi.tuna.tsinghua.edu.cn/simplepip install -e faster_coco_eval/ -ihttps://pypi.tuna.tsinghua.edu.cn/simplepip install -r requirements.txt -ihttps://pypi.tuna.tsinghua.edu.cn/simpleLvt-eval guidelinestudio aiportwrite your config files in "config/{your config files}.json".For example:
1) multi-object（studio json） vs. multi-object（interface return label） detection:
{
    "label":[{"objectLabel":["烟"], "attrLabel":[], "id":0, "prediction":"smoke"},{"objectLabel":["火"], "attrLabel":[], "id":1, "prediction":"fire"}],
    "aiport": "http://192.1.2.238:8893/vql/v1/serving/process",
    "rawdata": "data/yh.json",
    "draw": false,
    "download": false,
    "raw_prediction_path": "prediction_dirs/raw_predictions_8893.json",
    "save_gt_coco": "gt_dirs/coco_groundtruth.json",
    "save_pred_path": "prediction_dirs/prediction_results.json",
    "faster_coco_api": true
}

2) multi-object（studio json） vs. single-object（interface return label） detection:
{
    "label":[{"objectLabel":["人"], "attrLabel":["躺", "趴"], "id":0, "prediction":"睡岗"}],
    "aiport": "http://192.1.2.238:8324/vql/v1/serving/process",
    "rawdata": "data/sleep_test_json_0621.json",
    "draw": false,
    "download": false,
    "raw_prediction_path": "prediction_dirs/raw_predictions_8324.json",
    "save_gt_coco": "gt_dirs/coco_groundtruth.json",
    "save_pred_path": "prediction_dirs/prediction_results.json",
    "faster_coco_api": true
}

3) single-object（studio json） vs. single-object（interface return label） detection:
{
    "label":[{"objectLabel":["person"], "attrLabel":[], "id":0, "prediction":"person"}],
    "aiport": "http://192.1.2.238:8312/vql/v1/serving/process",
    "rawdata": "data/xingren.json",
    "draw": false,
    "download": false,
    "raw_prediction_path": "prediction_dirs/raw_predictions_8312.json",
    "save_gt_coco": "gt_dirs/coco_groundtruth.json",
    "save_pred_path": "prediction_dirs/prediction_results.json",
    "faster_coco_api": false
}"label":
    "objectLabel": enter the obj tags of studio json 
    "attrLabel": enter the attr tags of studio json (same the obj tags)
    "id": default is 0. 
    "prediction": label of original outputs of interface 
"aiport": model interface
"rawdata": studio json format data
"draw": "true" means draw GT and Pred bbox
"download": download images, if draw=True, need download=True
"raw_prediction_path": save original outputs of interface, file name "raw_predictions_{model interface id}.json"
"save_gt_coco": save studio json format to coco json format
"save_pred_path": save original outputs of interface to coco json format
"faster_coco_api": "true" means use faster coco evaluationthen runpythonod_evaluator.py--modestudio_json--config{yourconfigfiles}coco jsonprepare your data (coco json format)
prepare groundtruth and prediction json.then runpythonod_evaluator.py--modecoco_json--gt_json{yourgtcocoformatjson}--pred_json{yourpredcocoformatjson}orpythonod_evaluator.py--modefaster_coco_json--gt_json{yourgtcocoformatjson}--pred_json{yourpredcocoformatjson}How to start (example)use mode = studio aiport:
python od_evaluator.py --mode studio_json --config config/config_xingren.jsonuse mode = coco json:
python od_evaluator.py --mode coco_json --gt_json example_data/only_no_glove.json --pred_json example_data/only_no_glove_pred.jsonuse mode = faster coco json:
python od_evaluator.py --mode faster_coco_json --gt_json example_data/only_no_glove.json --pred_json example_data/only_no_glove_pred.jsonupdate pypi