Package: consnet
ConsNetThis repository maintains the official implementation of the paperConsNet: Learning Consistency Graph for Zero‐Shot Human‐Object Interaction DetectionbyYe Liu,Junsong YuanandChang Wen Chen, which has been accepted byACM Multimedia 2020.InstallationThe ConsNet package could be installed directly from PyPI or manually from source for different uses. Please refer to the following environmental settings that we use.CUDA 10.2 Update 2CUDNN 8.0.5.39Python 3.9.2PyTorch 1.8.1MMDetection2.11.0AllenNLP2.2.0NNCore0.2.4Install from PyPIYou may install ConsNet from PyPI and import it in your own project as a Python package. This library implements several useful functionalities includingPair IoU,Pair NMSandunified APIs for HICO-DET dataset.Simply run the following command to install the latest version of ConsNet.pip install consnetFor more details aboutconsnet.api, please refer to ourdocumentation.Install from sourceBy installing ConsNet from source, you may access the full capabilities of this project, including pooling object features, constructing the consistency graph and benchmarking the ConsNet model.Clone the repository from GitHub.git clone https://github.com/yeliudev/ConsNet.git
cd ConsNetInstall full dependencies and the package.pip install -e .[full]Getting StartedWe pre-extract the visual features of all the humans and objects in the dataset and save them for training as well as testing. These features are also used to construct the consistency graph. Please refer to our paper for more details about feature extraction and data sampling.Build dataset and construct the consistency graphDownload the checkpoints of object detector and ELMo.# Object detector checkpointswgethttps://dl.catcatdev.com/consnet/faster_rcnn_r50_fpn_3x_coco-26df6f6b.pth
wgethttps://dl.catcatdev.com/consnet/faster_rcnn_r50_fpn_20e_hico_det-77b91312.pth# ELMo options and weightswgethttps://dl.catcatdev.com/consnet/elmo_2x4096_512_2048cnn_2xhighway_5.5B_options.json
wgethttps://dl.catcatdev.com/consnet/elmo_2x4096_512_2048cnn_2xhighway_5.5B_weights.hdf5DownloadHICO-DETdataset and prepare the files in the following structure.ConsNet
├── configs
├── consnet
├── tools
├── checkpoints
│   ├── faster_rcnn_r50_fpn_3x_coco-26df6f6b.pth
│   ├── faster_rcnn_r50_fpn_20e_hico_det-77b91312.pth
│   ├── elmo_2x4096_512_2048cnn_2xhighway_5.5B_options.json
│   └── elmo_2x4096_512_2048cnn_2xhighway_5.5B_weights.hdf5
├── data
│   └── hico_20160224_det
│       ├── anno_bbox.mat
│       └── images
│           ├── train2015
│           └── test2015
├── README.md
├── setup.py
└── ···Convert the annotations to COCO format. The results will be saved indata/hico_det/annotations.python tools/convert_anno.pyBuild dataset and construct the consistency graph. The results will be saved indata/hico_det.python tools/build_dataset.py --checkpoint <path-to-checkpoint>Train a modelRun the following command to train a model using specified configs.python tools/launch.py --config <path-to-config>Test a model and evaluate resultsRun the following command to test a model and evaluate results.python tools/launch.py --config <path-to-config> --checkpoint <path-to-checkpoint> --evalCustomizationThanks to the modulized implementation based onNNCore, this project is highly customizable with a number of replaceable modules. You may play with the hyperparameters inconfigsor construct your own HOI detection pipeline by replacing the dataset, detector, embedder, etc. Please check thedocumentationof NNCore for more details about customizing the engine and modules.CitationIf you find this project useful for your research, please kindly cite our paper.@inproceedings{liu2020consnet,
  title={ConsNet: Learning Consistency Graph for Zero-Shot Human-Object Interaction Detection},
  author={Liu, Ye and Yuan, Junsong and Chen, Chang Wen},
  booktitle={Proceedings of the 28th ACM International Conference on Multimedia},
  pages={4235--4243},
  year={2020}
}