Weakly Supervised Knowledge Transfer with Probabilistic Logical Reasoning for Object Detection
Authors: Martijn Oldenhof
Adam Arany
Yves Moreau
Edward De Brouwer
Venue: ICLR 2023 poster (2023)
Keywords: weak supervision
knowledge transfer
object detection
probabilistic logical reasoning

Abstract: Training object detection models usually requires instance-level annotations, such as the positions and labels of all objects present in each image. Such supervision is unfortunately not always available and, more often, only image-level information is provided, also known as weak supervision. 
Recent works have addressed this limitation by leveraging knowledge from a richly annotated domain. However, the scope of weak supervision supported by these approaches has been very restrictive, preventing them to use all available information. In this work, we propose ProbKT, a framework based on probabilistic logical reasoning to train object detection models with arbitrary types of weak supervision. We empirically show on different datasets that using all available information is beneficial as our ProbKT leads to significant improvement on target domain and better generalisation compared to existing baselines. We also showcase the ability of our approach to handle complex logic statements as supervision signal.

Published as a conference paper at ICLR 2023
WEAKLY SUPERVISED KNOWLEDGE TRANSFER WITH PROBABILISTIC LOGICAL REASONING FOR OBJECT DE- TECTION
Martijn Oldenhof
Adam Arany ESAT STADIUS
ESAT STADIUS
KU Leuven, Belgium
KU Leuven, Belgium adam. wwwnyeeeat. kuleuven be
mart i jn. woannoldeen oldenhof@kuleeven be
Yves Moreau
Edward De Brouwer
ESAT STADIUS
ESAT STADIUS
KU Leuven, Belgium
KU Leuven, Belgium edward. debrouwez@gmail com
yves mww.eauesst.. www.llesss..kkkee....... kuleuven.
ABSTRACT
Training object detection models usually requires instance-level annotations, such the positions and labels of all objects present in each image. Such supervision is unfortunately not always available and, more often, only image-level information is provided, also known weak supervision. Recent works have addressed this limitation by leveraging knowledge from a richly annotated domain. However, the scope of weak supervision supported by these approaches has been very restric- tive, preventing them to use all available information. In this work, we propose ProbKTT, framework based on probabilistic logical reasoning that allows to train object detection models with arbitrary types of weak supervision. We empirically show on different datasets that using all available information is beneficial as our ProbKT leads to significant improvement on target domain and better generaliza- tion compared to existing baselines. We also showcase the ability of our approach to handle complex logic statements supervision signal. Our code is available at attts:://qttuue http:/hwwwwwnnneeeeeeeeeeeeeeeee
INTRODUCTION
Object detection is a fundamental ability of numerous high-level machine learning pipelines such autonomous driving [6]. augmented reality [42) image retrieval [17]. However, training state-of-the-ard object detection models generally requires detailed image annotations such as the box- coordinates location and the labels of each object present in each image. If several large benchmark datasets with detailed annotations are available [5), [5), providing such detailed annotation on new specific datasets comes with significant cost that is often not affordable for many applications.
More frequently, datasets come with only limited annotation, also referred to as weak supervision.
This has sparked research in weakly-supervised object detection approaches 25] 6 40) using
techniques such multiple instance learning 44 variations of class activation maps 3. However,
these approaches have been shown to significantly underperform their fully supervised counterparts
in terms of robustness and accurate localization of the objects 39
appealing and intuitive approach to improve the performance of weakly supervised object detection is to perform transfer learning from an existing object detection model pre-trained on a fully annotated dataset [14 46 43 This approach, also referred to as transfer learning or domain adaptation, consists in leveraging transferable knowledge from the pre-trained model (such bounding boxes prediction capabilities) to the new weakly supervised domain. This transfer has been embodied different ways in the literature. Examples include simple fine-tuning of the classifier of bounding box proposals of the pre-trained model [43], or an iterative relabeling of the weakly supervised dataset for retraining a new full objects detection model the re-lebeled data 46)

---

Published as a conference paper at ICLR 2023
Source Domain
Target Domain
Preds from pre- trained (on dom.)
Preds from pre- prainra (on S. dom.)
Preds after ProbKTT
Fiphers nihteer hurpe]
(cylinder, metal, small)
(cube, metal, smalll
nutte, langh
Englinoer mibluc.
Enpleree misil small
(eylinder, metal, large)
[ewbe, sosta,, largel
shapes
(cylinder, sphere)
shapes
fxube, cylinder, sphere)
material (metal, rahher)
material
(metal. rubher)
(small, large)
[small, large)
Y= hoxes: (lobels, positions))
Y=
(q (1 small metal cube. large metal cube, large nabber sphere, senall rubber cylinder))
Weakly supervised knowledge transfer with probabilistic logical reasoning. (Left) model can be trained on the source domain using full supervision (labels, positions) but only on limited of shapes (cylinders and spheres). (Middle) The pre-trained model does recognize the cubes from the target domain correctly. (Right) The model can adapt to the target domain after applying ProbKT and can recognize the cubes.
Figure I: ProbKT:
However, existing approaches very restrictive in the type of weak supervision they able to harness. Indeed, some do not support new object classes in the new domain [20), others can only use label indicating the presence of object class [46]. However, in practice, the supervision new domain can come in very different forms. For instance, the count of each object class can be given, such as in atom detection from molecule images where only chemical formula might be given. Or, when many objects are present on an image, a range can be provided instead of an exact class counts (e.g. "there are least cats this image"). Crucially, this variety of potential supervisory signals on the target domain cannot be fully utilized by existing domain adaption approaches.
the
To address this limitation, we introduce ProbKT, a novel framework that allows generalize knowledge transfer in object detection to arbitrary types of weak supervision using neural probabilistic logical reasoning
(27]. This paradigm allows to connect probabilistic outputs of neural networks with logical rules and to infer the resulting probability of particular queries. One can then evaluate the probability of a query such as "the image contains at least two animals" and differentiate through the probabilistic engine to train the underlying neural network. Our approach allows for arbitrarily complex logical statements and therefore supports weak supervision like class counts or ranges, among other. To knowledge, this is the first approach to allow for such versatility in utilizing the available information on the new domain.
To assess the capabilities of this framework, provide extensive empirical analysis of multiple object detection datasets. Our approach also supports any type of objects detection backbone architecture. We thus two popular backbone architectures, DETR [7] and RCNN 34] and evaluate their performance in terms of accuracy, convergence as well of generalization on out- oftto--diitribution data. Our experiments show that, due its ability use the complete supervisory signal, our approach outperforms previous works in wide range of setups.
Key contributions: (1) We propose novel knowledge transfer framework for object detection relying on probabilistic programming that uniquely allows using arbitrary types of weak supervision on the target domain. (2) We make our approach amenable to different levels of computational capabilities by proposing different approximations of ProbKT. (3) We provide an extensive experimental setup to study the capabilities our framework for knowledge transfer and out-of-distribution generalization.
RELATED WORKS
comparative summary of related works is given in Table [ We distinguish three main categories: (1) pure weakly supervised object detection methods (WSOD) that do not leverage richly annotated source domain, (2) unsupervised object detection methods with knowledge transfer (DA or domain adaptation methods) that do not supervision the target domain and (3) weakly supervised
2

---

Published as a conference paper at ICLR 2023
object detection methods with knowledge transfer (WSOD w/transfer) that are restrictive in the type of supported weak supervision. To our knowledge, our work is the first to allow for arbitrary supervision the target domain (and supporting new classes in the target domain) while also leveraging knowledge from richly annotated domains. ProbKTT supports arbitrary weak supervision thanks to the inherited expressiveness of Prolog [4 which is based on subset of first-order predicate logic, Horn clauses and is Turing-complete.
Weakly supervised object detection (WSOD) This class of method allows training object detection models with only weak supervision. One can thus train these approaches directly on the target domain. However, they do not allow to leverage potentially available richly annotated datasets, which has been shown to lead to worse performance [39]. Different flavors of WSOD architectures have been proposed relying on variety of implementations such as multiple instance learning (MIL))-ased 25,40 or class activation (CAM) based [47 3 In contrast to WSOD methods, our approach is designed to exploit existing richly annotated datasets and thus provides increased performance on the target domain. For comprehensive review of WSOD methods we refer the reader to Shao et al. [39]
Domain adaptation methods (DA) In contrast to WSOD methods, domain adaptation methods do rely on fully supervised source domain dataset. However, they do not assume any supervision on the target domain and are therefore not equipped to exploit such signal when available [37 8 22 48).
WSOD with knowledge transfer Our approach belongs to the class of weakly supervised object detection models with knowledge transfer. These methods aim to transfer knowledge from source domain, where full supervision is available, target domain where only weak labels are available. Existing work in this class of models only allows for limited type of supervision of the target domain. Most architectures only support label indicating the presence or absence of class of object in the imagel [4: 46 433. Inoue et al. (00 allows for class counts as weak supervision but unfortunately does not allow for new classes the target domain. In contrast, ProbKT natively allows for class counts and new classes as well as other types of weak supervision.
Neural probabilistic logical reasoning Probabilistic logical reasoning combines logic and probability theory. Favored for its high-level reasoning abilities, it was introduced as an alternative way to deep learning the quest for artificial intelligence [0]. Statistical artificial intelligence S2 23 and probabilistic logic programming [II] are examples of areas relying on these premises. In unification effort, researchers have proposed hybrid architectures, embedding both deep learning and logical reasoning components (38] 35 Our work builds upon the recent advances in the field, where combinations of deep learning, logical, and probabilistic approaches were introduced (27). allowing high-level reasoning with uncertainty using differentiable neural network architectures.
Method
Type
Amsocated source dom.
Weak supervision
New classes
Implementation
Li al. 25
WSOD
X
presencelsbsencce presencelabsennee presencolabsence
MIL bhsed
Bilen and Vedaldi
WSOD
spatial pyramid pooling layer
Song at
MIL based
Zhou al. 4x
WSOD
X
mix mix
CAM bused
Bac 8
WSOD
X
CAM based
Kundu ct
74
DA
one-shox
Intre-nncremental DA
al. [7]
Strong Beong-Weak Distrbbution Alignment
Chen al.
Adverstrial maining
Kim al. [2]
DA
X
Adversatial training and Domain Diversifiation
Zhu et al. 18
DA
X
solective region adiptation framework
Descluers al.
WSOD wrramsferr
presenc/absence
CRF-bued,, iteratively
Zhong
prssenclabseenee
based, fteratively
Uijlings
WSOD witransfer
pessenellsseence
MII. hased, iteratively
Inone et al. 20
WSOD witransfer
class counts
DA paeudolabeling inerntively
ProbKT (ours)
WSOD witranster
arhitrary
Probahilistic lagical reasoning, itenativlly
Table I: Summary table of related works with weakly supervised object detection (WSOD), Domain Adaptation(DA) and weakly supervised knowledge transfer methods (WSOD w/ transfer).
3 METHODOL.OGY
3.1 PROBLEM STATEMENT
We consider the problem of weakly supervised knowledge transfer for object detection. Using model trained on a richly annotated source domain, we aim at improving its performance on a less richly annotated target domain.
3

---

Published as a conference paper at ICLR 2023
Let De ( vL) :1= :i=,;.,,N)N be a dataset issued from the source domain and consisting of Ng images Ig along with their annotations. We write by E R2; x4 and y E fI...,K, )"; for the box coordinates and class labels of objects in image I n, is the number of objects present in image 13 and K₂ is the total number of object classes in the source domain. This represents the typical dataset required to train classical fully-supervised object detection architectures. The target dataset D. = {(1)) gf) : ..., ] contains N₁ image from the target domain along with image-level annotations qi- These annotations are logical statements about the content of the image in terms of object classes and their location. Examples include the presence of different classes in each image (i.e. the classical assumption in weakly supervised object detection) but also extends to the counts of classes or complex combination of counts of objects attributes (e.g. "two red objects, and at least two bicycles"). What more, the logical statements of can include classes not already present the source domain. This type of logical annotation is then strictly broader than the restrictive supervision usually assumed.
Based on the availability of a source dataset and target dataset as described above, our goal is then to harness the available detailed information from the source domain to perform accurate object detection the target domain. A graphical illustration of this process is given in Figure []
3.2 BACKGROUND
3.2.1 OBJECT DETECTION
Object detection aims at predicting the location and labels of objects in images. One then wishes
to learn a parametric function fe : I + {B x Rx3² with fo(1) = {b. (y)) {br, Py.i) .) :
1, .... f such that the distance between predicted and true boxes and labels, d({ ((.( {(b, ((b,))* ") is minimum. Objects detection architecture would usually output box features proposals (h: :
1...., A conditioned on which they would predict the prohability vector of class labels py,i 9p (ha)
and the box location predictions b, = 9b (h₂) using shared parametric functions gp(-) and g-(-). For an object n, we write the predicted probability of the object belonging to class h as py...
3.2.2 PROBABILISTIC LOGICAL REASONING
Probabilistic logical reasoning uses knowledge representation relying on probabilities that allow encoding uncertainty in knowledge. Such a knowledge is encoded in probabilistic logical program P as set of N probabilistic facts U = (U1,..., UN} ) and M logical rules F = (J,.-..MM) connecting them. A simple example of probabilistic fact is "Alice and Bob will each pass their exam with probability 0.5" and an example of logical rule is "if both Alice and Bob pass their exam, they will host a party". Combining probabilistic facts and logical rules, one can then construct complex probabilistic knowledge representation, that can also be depicted as probabilistic graphical models.
Probabilistic logical programming allows to perform inference by computing the probability of a particular statement or query. For instance, one could query the probability that "Alice and Bob will host a party". This query is executed by summing over the probabilities of occurrence of the different worlds w = (ui...., uN) } (i.e. individual realization of the set of probabilistic facts) that are compatible with the query The probability of a query q in program P can then be inferred as Pr(g) Σ.. P(w)- [[F(ww) q]. where F(w) q stands for the fact that propagation of the realization w across the knowledge graph, according to the logical rules F leads to being true.
Remarkably, recent advances in probabilistic programming have lead to learnable probabilistic facts (27] In particular, the probability of a fact can be generated by a neural network with learnable weights. Such a learnable probabilistic fact is then referred to as neural predicate Uº, where we make the dependence the weights @ explicit. One can then train these weights to minimize a loss that depend on the probability of a query q: 0 arg ming [(P(q) 8))-
Our approach builds upon this ability to learn neural predicates and uses DeepProbLog 27 as the probabilistic reasoning backbone. DeepProbLog is neural probabilistic logic programming language that allows to conveniently perform inference and differentiation with neural predicates. We refer the reader the excellent introduction of Manhaeve et al. (88 for further details about this framework.
4

---

Published as a conference paper at ICLR 2023
3.3 PROBKT: WEAKLY SUPERVISED KNOWLEDGE TRANSFER WITH PROBABILISTIC LOGICAL REASONING
graphical description of our approach is presented in Figure Our framework starts from pre-trained object detection model fe on the source domain. The backhone of this model is extracted and inserted into new object detection model fa with new target box position predictors and box label classifiers. This new model is then used to predict box proposals along with the corresponding box features on target domain images 7,. These box features are then fed new target box position predictor and box label classifier. The predictions of this classifier are considered neural predicates and are given to probabilistic logical module. This module evaluates the probability of queries gt. the loss, and the corresponding gradient that can be backpropagated to the classifier and the backbone. As we want to maximize the probability of the queries being true, we use the following loss function:
Cg Σ (INN))DD
-logPola (1))
In theory, the backbone can be trained end end with this procedure. Our experiments showed that only updating the box features classifiers resulted in more stability as also shown in previous works [46]. We then adopt here the same iterative relabeling strategy, as described next.
Co(Pr(g) P..
Probabilistic Reasoning
Av,
b0
S.11
Pa.a
py Labels probabiliies
+
Box positions
h Box features
ho
h1
hp
Probabilistic query
Object Detection Backbone
fū model.
Image
Figure 2: ProbKT. The pre-trained object detection backbone outputs the box features h for the detected objects. Box classifiers (red) and box position predictors (blue) then predict corresponding label predictions Py and box position predictions b that are fed to the probabilistic reasoning layer. This layer computes the probability of the query along with the gradients with respect to py and that can be backpropagated through the entire network.
3.3.1 ITERATIVE RELABELING
The approach described above allows to fine-tune our model fü to the target domain. To further improve the performance, propose iterative relabeling strategy that consists in multiple steps fine-tuning, re-labeling and re- re-training. A similar has also been proposed by Zhong et al. [46].
Fine-tuning. This step corresponds to training ProbKT on the weakly supervised labels, by minimizing the loss of Equation
Re-labeling, Once ProbKT has been trained,
its predictions to annotate images in the
target domain. In practice, we only relabel images for which the model predictions comply with the available query labels in order avoid too noisy labels.
Re-training, The re-labeled target domain can used to re- re-train the object detection backbone of ProbKTin fully-supervised fashion.

---

Published as a conference paper at ICLR 2023
This procedure can be repeated multiple times to improve the quality of the relabeling and the quantity of relabelled in the target domain dataset. graphical representation of the relabeling pipeline is presented in Figure
Pre-training
Fine-tuning
Re-labeling
Re-training
Iterative relabeling. full cycle is composed a fine-tumin fine-tuning, a re-labeling and a re- re-training step. After eycle, the fine-tuning step and/or re-labeling step be iteratively repeated.
Figure 3:
3.3.2 COMPUTATIONAL COMPLEXITY AND APPROXIMATIONS
The computational complexity of inference probabilistic programming depends on the specific query q and several approximations have been proposed for improving the computation time (44] We propose two approaches for reducing the computational cost adapted to object detection: (1) filtering the data samples before applying ProbKT (see Appendix Section C.TD or (2) when the supervision consists of the class labels counts, considering only the most probable world (ProbKT*) instead of all possible worlds.
3.3.3 PROBKT*; THE MOST PROBABLE WORLD AND CONNECTION TO HUNGARIAN MATCHING
The probabilistic inference step requires a smart aggregation of all worlds compatible with the query q. Yet, in certain cases, one can reduce the computational world. Indeed, consider the case when the query consists of the list of different class labels in the images. For number of boxes á proposed by the objects detection model, the query can be written as the set of labels 9 = {y' : = 1. .... A If we further write py, as the probability of the label of box n belonging to class lo given by the model (as introduced in Section 3.2.1. we have:
by only considering the most probable
a!
Pp(g)
Σ Py. P
(0)
(1)
(R) P
EI (n) v.m
V.A
j=1
where oj corresponds to the jth permutation of the query vector q. To avoid the computation of each possible world contribution, one can only use the configuration with the largest contribution Pp(4) and discard the other ones.
This possible world corresponds to the permutation a² that satisfies:
0 arg max log(
Pg.n (g)(m))
(m) = arg max Σ P = arg min I - pas (1)(()))
Remarkably, this corresponds the solution of the best alignment using the Hungarian matching
algorithm with cost c(n) = (1 (ggh as used, among others, in DETR 77. Thus, when the query is is the set of class labels, the most plausible world can thus be inferred with the Hungarian
matching algorithm. In Appendix C.2] we also show that the gradient of ProbKT can be interpreted as probability weighted extension of the gradient resulting from Hungarian matching.
EXPERIMENTS
4.1 DATASETS
We evaluate approach on three different datasets: (1) CLEVR-mini dataset, (2) Molecules dataset with images of chemical compounds, and (3) an MNIST-based object detection dataset. For each dataset, three subsets, corresponding different domains, used: (1) source domain, (2) a target domain, and (3) an oft-of-distribution domain (OOD). The source domain the richly annotated domain that was used pre-train the object detection model. The target domain is

---

Published as a conference paper at ICLR 2023
the domain of interest but with image-level annotations only. Lastly, the OOD domain contains images from a different distribution than the source and target domains and is used to study the generalizability of the models. Source and target domains are split into 5 folds of train and validation sets and an independent test set. We focused our experiments on the small sample regime (range 1k-2k numbers of samples) both for the source the target domain. More details each dataset can be found in Appendix B
4.2 MODELS
In the experiments, we apply our method ProbKT on two different pre-trained object detection backbone models: (1) DETR 7 and (2) FasterRCNN 34 Both are pre-trained on the COCO dataset [26]. We also evaluate an Hungarian-algorithe approximation (ProbKT**) of our method when the weak supervision allows it. For sake of conciseness, we omit the results of ProbKT* here but they can be found in Appendix D The details of the training procedures, as well as the hyper-parameters used for the different models and the different datasets summarized in Table 4 in Appendix [A]
4.2.1 BASELINE MODELS
As shown in Section 2 all available approaches for weakly supervised object detection are very restrictive in terms of the supervision signal they support. Our main comparison partner is the state of the art WSOD-transfe method [46).
Additionally. we compare our approach against a Resnet50 [8] backbone pre-trained on Ima- geNet [2]. Fine-tuning is performed by adding an extra multitask regression layer that is trained to predict the individual counts of the objects in the image as in Xue et al. [45]. This architecture natu- rally relies only on label counts in the target images for fine-tuning. We then predict box predictions using class activation maps as in Bac et al. 3 to compare its performance on object localization. We call this approach Resmer50-CAn
When the supervision signal allows it, we also compare with a DETR model trained end-to-ent jointly target and source domains, masking the box costs in the matching cost of the Hungarian algorithm for image-level annotated samples. We call this approach DETR-joint.
Model
Data Domain
CLEVR count avc.
CLEVR mAP (mAP@1oU=0.5)
Mol. count. ace
Mol. mAP (mA.@)0U=00.)
Resmet50.CAM
target domain
0.97 + 0.005
0.036 = 0.014 (0.200 = 0.071)
0.978 # 0.004
0.0 0.0 (0
Resmet50.CMM CAM
OOD
0,831 1: 0.018
0.029 2 0.010 (0.153 a 0.041)
0.0 =0
m/2"
Resmet50 CAM
source domnin
0.019 (0.178 0.064)
0.021
0.0
WSOD-transee
Iarget domain
0.944 + 0.004
0.844 + 0.005 (0:988 0.001)
3.001 + 0.0
0.018 + 0.004 (0.061 0.011)
WSOD
DOD
0.73 + 0.011
0.79 0.006 (0.900 =0.001)
0.003 = 0.002
m/a"
WSOD transfer
domain
0.959 4: 0.001
0.926 0226200000 0.001 (0.995 0.0)
0.0
0.021 0.003 (0.009 0.009)
DETR-join
target domain
0.150 + 133
0.57U 0.010.11222 (0.664
(.018)
0.357 + 0.196
0.107 :: (0045 0.071)
DETR. DRTR-joiit
D0D
0.084 + 0.089
0.534 + 0.012 (0:66 - (.012)
0.024 + 0.021
n/a"
DETR-joint
doen
0,923 1: 0.019
0.908 1: 0.017 (0.992 :: 0.001)
0 0,232 :: 0.127
0.23 1: 0.063 (0.565 0.08)
(pre-trinool)
larget DOD
0.011
(0.785 0.004)
RCNN (pre-trained)
0.582 :: 0.012 (0.603 0.011)
0.348 = 0.096
n/a"
RCNN (pre-traincd)
souree domain
0.988 1: 0.002
0,984 1:: 0.01 (0.996 0.0)
0.948 1: 0.004
0.737 :: 0.005 (0.979 =t: 0.0)
DETR tra-nedi
lurget domain
0.0 4 n..
0.498 0.619 (0.533 1 0.024)
0,164 1: 0.033
0.314 1 0.006 (0.542 (0.06))
DETR (pre-trinne))
0.001
DETR (pre trained)
source domnain
0.97 1: 0.009
0.945 :: 0.009 (0.992 1: 0.001)
0.581 =: 0.022
0.409 4: (0.000550..722222 0.004)
ProbKT (DRTR)
target domain
0.946 + 0.014
0.803 0.001 (0:980 (0006)
0.027
0.204 + 0.02 (0.507 0.014)
ProbKT (DETR)
0OD
0.726 + 0.005
0.715 + 0.000 (0:974 + 0.006)
0.004 = 0.003
n/a"
ProbKT (DETR)
source domain
0.987 4: 0.003
0.948 4: 0.005 (0:995 4: 0.001)
0.549 : 0.026
0.38 4: (0.733 0.006)
PreкT (RCNN)
target domain
0.975 0.003
0.856
(0.983 0.001)
0.299 U.041 (0.829 0.054)
ProSKT (RCNN)
DDD
(188 0.022
0.833 0.042 (0.991 = 0.001)
0.603 = 0.037
mia""
ProbKT (RCNN)
source domain
0.995 d: 0.002
0.941 + 0.041 (0:908 + 0.001)
0.96 0.002
0.666 + 0.005 (0.978 0.002)
Table 2: Results of the experiments for the datasets: CLEVR-mini and Molecules. Reported test accuracies over the 5 folds. Best method is in bold for each metric and data distribution. *: OOD test set of Molecules dataset has no bounding box labels.
4.3 EVALUATION METRICS
We evaluate the performance of the models on the different datasets based on two criteria the count accuracy and the objects localization performance. The count accuracy measures the ratio of correct images where all individual counts of (all detected) objects are correct. To evaluate how well the
7

---

Published as a conference paper at ICLR 2023
model is performing in localizing the different objects in the image report the mean average precision (mAP) performance, a widely used metric for evaluating object detection models.
4.4 WEAKLY SUPERVISED KNOWLEDGE TRANSFER WITH CLASS COUNTS
We first investigate the performance of ProbKT when the weakly supervision consists of class counts only. The query g for each image then consists of the number of objects from each class in the image. We evaluate the models on the CLEVR-mint and Molecules datasets. For the Molecules dataset, the query for image containing 6 carbon atoms (C), 6 oxygen atoms (O) and 12 hydrogen atoms (H) result in the following query: g = ((C, O, H]. [6, 6, 12).. These weak labels in the case of the Molecules dataset are widely and easily available in the form of the chemical formula of the molecule on the image (e.g Cg (1))). The recognition of atomic level entities on images of molecules is a challenge in the field of Optical Chemical Structure Recognition (OCSR) p 33 [9] For the CLEVR-mini dataset, the query for an example image containing 2 spheres, I cylinder and 3 cubes would be q = ([Cube, Cylinder, Sphere], [3, 1, 2,). Formal descriptions of the queries for each task
are presented in Appendix E
Results of the experiments are summarized in Table We observe on both datasets that ProbKT is able to transfer knowledge from the source domain to the target domain and improve count accuracy on the target domain and most cases also on the source domain. The count accuracy increases on both the target domain and on OOD, suggesting better generalization performance. This is in contrast with Resne150-CAM which performs well on the target domain of the Molecules dataset but fails on OOD. We also note significant improvement in object localization (mAP) for ProbKT on the CLEVR-mini dataset. However, fine-tuning seems detrimental for mAp on the Molecules dataset. This can be explained by the very small bounding boxes in the Molecules dataset. We therefore also report the mAP@ IoU=0.5 where we observe some increase in performance after tieettunigg Lastly, we observe that our approach outperforms WSOD-transfer on all metrics for both datasets. wSOD-transfer performs well on CLEVR-mini but fails for the Molecules dataset. This can be explained by the fact that this method only supports class indicators (whether a class present in the image), which is particularly detrimental in molecules images containing a lot of objects.
4.5
OTHER TYPES OF WEAK SUPERVISION
4.5.1
CLASS RANGES
The annotation of images a tedious task, which limits the availability of fully annotated datasets. When the number of objects on an image is large, counting the exact number of objects of a particular class becomes too time-consuming. A typical annotation in this case consists oof class ranges where, instead of exact class counts, an interval is given for the count. For example an image from the CLEVR-mini dataset with more than 4 cubes, exactly 4 cylinders and less than spheres would result in the following query: g = ([cube, cylinder. sphere], [[4, ( [4, 5[. [0, 4[].. We evaluate this experimental setup and report results in Table 3 We observe that ProbKT performs significantly better than WSOD-transfer
count accuracy, which still uses only presence/absence labels. We note that Resnet50-CAM is unable to use this type of supervision and is thus reported as n/a.
MNIST (mAPe) ()
CLEVE" CLEVIC (mAPWWot(sn))))
Rese150-CAM
damsin
3,064 0.041
0.506 & 01005
0.000 0.000/0.0:4
Resuc150 CAM
OOD
3U01
0.015 à 11,004
0.002)0.011
Resec150 CAM
domain
0.127 =
0,540
0.10G 0,00400.028 0,018) s/s
WSOD innderr
Barge domain
nla
aia
0.944 + 0.004
0.844 + 0.005 (0.088 1: 0,001)
WS0D insser
OOD
s/a
1.73 2: 0.011
0,79 = 2U005 (1.669 1: (1,01)
WSOD inaste
scucce domalo
Bla
n/a
1,360 à 0,001
0,326 0,001 (0.936 0.01)
RCNN (pre-tmined)
Earget donsin
02992 CLU5
0,205 e D.0015
(1832 I 0.01 (9-885 i (000))
0.0 0.0
0.5e6 0.01 00-088 it Q.033
RCNN (pm-tmined))
0.005
0000
0.212 + 0.001
(.05) 0.0 (0 + 0.007)
0:0 0.0
0.582 0.0 2 (00000 + 0U11)
RCNN (pm-tnincal)
sarre domnin
0.90
- 0.008
0:901 + 0.005
0.917 - 0.021 (0.965 0.002)
0.985 + 0.002
0.984 + 0.01 (0.900 + 0.00
PrbkT (RCNN)
terpet damain
0.902-1 0.005
0.905 005
0.768 0.021 (0. (0.974 0.001)
0.971
0.006
0. 1.838 0.004 (0.983 0.001)
PoAKT (RCNN)
OOD
16863-
0.008
0.865 + 008
0.778 = 921 (0.97 + 0.000)
0.884 1:544-000 0.01
0.812 + 0.056 (0.991 + 0.001)
PrAKT (RCNN)
Jonuzin
0.967
0.004
0.967 0.00
0.873 0.036 (0.989 0.0013
0.994 + 0.001
0,322 # 0.035 (0.998 0.001)
Table Results of the experiments on the MNIST object detection dataset and on CLEVR* * dataset (*CLEVR uses ranges of class counts as labels instead of exact class counts). Reported test accuracies over the folds. Best method is bold for each metric and data distribution.
8

---

Published as a conference paper at ICLR 2023
4.5.2 COMPLEX QUERIES
More complex types of weak supervision than the ones considered above are also possible. To illustrate the capabilities of our approach, we build an MNIST object detection dataset where images show multiple digits as objects. Examples images are available in Appendix The weak supervision is here the sum of all digits in the image: 9 SUM(digito) Our ProbKT can seamlessly integrate this type of supervision as shown in Table 3 As all other baselines are unable process this type of supervision, we compare against a pre-trained RCNN and variation of Resnet50-CAM where we add an extra neural network layer that sums the individual counts to give the resulting sum. We report count accuracy, mAP and sum accuracy. The sum accuracy measures the ratio of correct images where the predicted sum (instead of the label of the digits) is correct. Details about the results on extra experiments with DETR backbone using complex types of weak supervision can be found in Appendix D
4.6
ABLATION STUDIES
CLEVR Dataset Performance
1.0
Performance
1.0
Dataset Perforaance
1.2
Count Accuracy
0.8
0.6-
0.4
(hrbbe
0.4
RCON (ProbiC)
Sum Val.
0.4
0.2
0.
(ProbiCH
(PrbbTT)
0.2
ACNN [ProkTi
0.3
(PrbbiT9)
DETR (Pobit)
Iteration
Itoration
(a) CLEVR iterative relaheling
(b) Molecules iterative relaheling
(c) MNIST iterative relabeling
Figure 4: Iterative relabeling performance for the different datasets. Iteration O: pretrained on source domain. Iteration 1: fine-tuned. Iteration 2: re-labeled and re-trained. Iteration 3: relabeled and re-trained. Iteration 4: relabeled and re-trained.
Iterative relabeling. In Figure we plot the evolution of the performance on the test sets after multiple rounds of fine-tuning and re-labeling, as detailed in Section [.3.T The final performance reported in the results tables is selected based on best relabeling iteration on the validation dataset. We observe that iterative relabeling after fine-tuning can improve performance significantly. Nevertheless, the benefit of iterative relabeling is less pronounced for DETR the Molecules dataset. We impute it to the fact that the fine-tune -uned DETR model is less accurate on this dataset.
Object detection backbone
Our method can scamlessly accommodate different object detection backbones. In Table 2 we present the results for our method with DETR and FasterRCNN backbone. We observe that PasterRCNN is typically performing better. In particular, the DETR backbone performs poorly on the Molecules dataset. This could be due the small objects in the Molecules dataset. Indeed, Carion et al. 7 recommend to use DETR-DG or DETR-DC5-R100 for small objects instead.
5 CONCLUSIONS AND DISCUSSION
Objects detection models are a key component of machine learning deployment in the real world. However, training such models usually requires large amounts of richly annotated images that are often prohibitive for many applications. In this work, we proposed novel approach to train object detection models by leveraging richly annotated datasets from other domains and allowing arbitrary types of weak supervision on the target domain. Our architecture relies on probabilistic logical programming engine that efficiently blends the power of symbolic reasoning and deep learning architecture. As such, our model also inherits the current limitations from the probabilistic reasoning implementations, such as higher computational complexity. We proposed several approaches to speed-up the inference process significantly and work will directly benefit from further advances in this field. Lastly, the versatility of probabilistic programming could help support other related tasks in the future, such image to graph translation.

---

Published as a conference paper at ICLR 2023
Reproducibility Statement Details for reproducing all experiments shown in this work available in Appendix More details on the datasets used in the experiments can be found Appendix B
ACKNOWLEDGMENTS
AA, MO and YM are funded by (1) Research Council KU Leuven: Symbiosis (C14/22/125), Symbiosis3 (C14//1/092); (2) Federated Bloudbasee Artificial Intelligence-driven platform for liquid biopsy analyses (C3/20/100); (3) CELSA Active Learning (CELSA/21/019); (4) European Union's 's Horizon 2020 research and innovation programme under the Marie Sklodowska-Curt grant agreement No. 956832; (5) Flemish Government (FWO: SBO (S003422N), Elixir Belgium [000289N]] SB and Postdoctoral grants: S003422N, ISB272IN, 1898819N, 12Y5623N) and (6) VLAIO PM: Augmenting Therapeutic Effectiveness through Novel Analytics JHBC.2019.25208 (7) YM, AA, EDB, and MO affiliated to Leuven.Al and received funding from the Flemish Government (AI Research Program). EDB is funded by FWO-SB grant (S98819N), Computational resources and services used in this work were partly provided by the VSC (Flemish Supercomputer Center), funded by the Research Foundation Flanders (FWO) and the Flemish Government department EWI.
REFERENCES
[1]
Mnist object detection dataset.
URL https:://gitlub.ccom/uukke
MNI Ww..ST--Obeecttttectiiio accessed on 01.02.2022.
[2] Rdkit: Open-source cheminformatics. URL https://wwwrrkie
accessed on
01. 02..22202..
[3] Wonho Bae, Junhyug Noh, and Gunhee Kim. Rethinking class activation mapping for weakly supervised object localization. European Conference on Computer Vision, pages 618 634. Springer, 2020.
[4] Aseem Behl, Omid Hosseini Jafari, Siva Karthik Mustikovela, Hassan Abu Alhaija, Carsten Rother, and Andreas Geiger. Bounding boxes, segmentations and object coordinates: How important is recognition for 3d scene flow estimation in autonomous driving scenarios? In Proceedings of the IEEE International Conference on Computer Vision, pages 2574 2583, 2017.
[5] Lukas Biewald. Experiment tracking with weights and biases, 2020. URL httph:t////// www wandb. comm Software available from wandb.com.
[6] Hakan Bilen and Andrea Vedaldi. Weakly supervised deep detection networks. In Proceedings of the IEEE Conference on Computer Vision and Patlern Recognition, pages 2846-2854, 2016.
[7] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European conference computer vision, pages 213-229. Springer, 2020.
[8] Yuhua Chen, Wen Li, Christos Sakaridis, Dengxin Dai, and Luc Van Gool. Domain adaptive faster r-cnn for object detection in the wild. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 3339-3348, 2018
[9] Djork-Améé Clevert, Tuan Le, Robin Winter, and Floriane Montanari. Img2mol- accuraae smiles recognition from molecular graphical depictions. Chemical science, 12(42): 2021.
[10] Lue De Raedt and Kristian Kersting. Probabilistic logic learning. ACM SIGKDD Explorations Newsletter, 5(1):31-45 2003.
[11] Luc De Raedt and Angelika Kimmig. Probabilistic (logic) programming concepts. Machine Learning, 100(1):5-47, 2015.
[12] Jia Deng, Wei Dong. Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hicrarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248-28 255. lece, 2009.
10

---

Published as a conference paper at ICLR 2023
[13] Li Deng. The mnist database of handwritten digit images for machine learning research [best of the web]. IEEE signal processing magazine, 1916)114 2012.
[14] Thomas Deselaers, Bogdan Alexe, and Vittorio Ferrari. Weakly supervised localization and learning with generic knowledge. International journal of computer vision, 100(3):255-293, 2012.
[15] M. Everingham, L. Van Gool, C. K. I. Williams, J. Winn, and A. Zisserman. The pascal visual object classes (voc) challenge. International Journal of Computer Vision, 88(2):303-338, June 2010.
[16] Elconora Giunchiglia, Mihaela Cätälina Stoian, Salman Khan, Fabio Cuzzolin, and Thomas Lukasiewicz. Road-r: The autonomous driving dataset with logical requirements. arXiv preprint arXiv:2210.01597. 2022.
[17] Ibtihaal M Hameed, Sadiq H Abdulhussain, and Basheera M Mahmmod. Content-based image retrieval: A review of recent trends. Cogent Engineering, 8(1):1927469, 2021.
[18] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[19] Rodrigo Hormazabal, Changyoung Park, Soonyoung Lee, Schui Han, Yconsik Jo, Jaewan Lee, Ahra Jo, Seung Hwan Kim, Jaegul Choo, Moontae Lee, al. Cede: A collection of expert- curated datasets with atom-level entity annotations for optical chemical structure recognition.
[20] Naoto Inoue, Ryosuke Furuta, Toshihiko Yamasaki, and Kiyoharu Aizawa. Cross-domain weakly-supervised object detection through progressive domaín adaptation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5001-5009, 2018.
[21] Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Clevr: diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 2901-2910, 2017.
[22] Taekyung Kim, Minki Jeong, Seunghyeon Kim, Seokeon Choi, and Changick Kim. Diversify and match: A domain adaptive representation learning paradigm for object detection. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 12456-12465, 2019.
[23] Daphne Koller, Nir Friedman, Sašo Džeroski, Charles Sutton, Andrew McCallum, Avi Pfeffer, Pieter Abbeel, Ming-Fai Wong, Chris Meck, Jennifer Neville, al. Introduction to statistical relational learning. MIT press, 2007.
[24] Jogendra Nath Kundu, Rahul Mysore Venkatesh, Naveen Venkat, Ambareesh Revanur, and Venkatesh Babu. Class-incremental domain adaptation. In European Conference on Computer Vision, pages 53-69. Springer, 2020.
[25] Dong Li, Jia-Bin Huang, Yali Li, Shengjin Wang, and Ming- Hsuun Yang. Weakly supervised object localization with progressive domain adaptation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3512-3520, 2016.
[26] Tsung-Y Yi Lin, Michael Maire, Serge Belongic, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pages 740-755. Springer, 2014.
[27] Robin Manhaeve, Sebastijan Dumancic, Angelika Kimmig, Thomas Demeester, and Luc De Raedt. Decpproblog: Neural probabilistic logic programming. Advances in Neural Informa- tion Processing Systems, 31, 2018.
[28] Robin Manhaeve, Sebastijan Dumančić, Angelika Kimmig, Thomas Demeester, and Luc De Raedt. Neural probabilistic logic programming in deepproblog. Artificial Intelligence, 298: 103504, 2021.
11

---

Published as a conference paper at ICLR 2023
[29] Martijn Oldenhof, Adam Arany. Yves Moreau, and Jaak Simm. Chemgrapher: optical graph recognition of chemical compounds by deep learning. Journal of chemical information and modeling, 60(10):4506-45) 45177, 2020.
[30] Martiju Oldenhof, Adam Arany, Yves Moreau, and Jaak Simm. Self-labeling of fully mediating representations by graph alignment. In Benelux Conference Artificial Intelligence, pages 46-65. Springer, 2021.
[31] Martijn Oldenhof, Ádám Arany, Yves Moreau, and Edward De Brouwer. Updating object detection models with probabilistic programming. 2022. ICML BeikshopUUpMi
[32] Lue De Raedt, Kristian Kersting, Sriraam Natarajan, and David Poole. Statistical relational artificial intelligence: Logic, probability, and computation. Synthesis lectures artificial intelligence and machine learning, 10(2):1-899, 2016.
[33] Kohulan Rajan, Achim Zielesny, and Christoph Steinbeck. Decimer: towards deep learning for chemical image recognition. Journal of Cheminformatics, 12(1))1-9, 2020.
[34] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. Advances in neural information processing systems, 28, 2015.
[35] Tim Rocktäschel and Sebastian Riedel. End-to-end differentiable proving. Advances in neural information processing systems, 30, 2017.
[36] Noureddin M Sadawi, Alan P Sexton, and Volker Sorge. Chemical structure recognition: rule-based approach. In Document Recognition and Retrieval XIX, volume 8297, page 82970E. International Society for Optics and Photonics, 2012.
[37] Kuniaki Saito, Yoshitaka Ushiku, Tatsuya Harada, and Kate Saenko. Strong-weak distribution alignment for adaptive object detection. Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pages 6956-6965, 2019.
[38] Adam Santoro, David Raposo, David G Barrett, Mateusz Malinowski, Razvan Pascanu, Peter Battaglia, and Timothy Lillicrap. A simple neural network module for relational reasoning. Advances in neural information processing systems, 30, 2017.
[39] Feifei Shao. Long Chen, Jian Shao, Wei Ji, Shaoning Xiao, Lu Ye, Yueting Zhuang, and Jun Xiao. Deep learning for weakly-supervised object detection and localization: survey. Neurocomputing, 2022.
[40] Hyun Oh Song, Ross Girshick, Stefanie Jegelka, Julien Mairal, Zaid Harchaoui, and Trevor Darrell. On learning to localize objects with minimal supervision. In International Conference on Machine Learning, pages 1611-1619. PMLR, 2014.
[41] Leon Sterling and Ehud Y Shapiro. The art of Prolog: advanced programming techniques. MIT press, 1994.
[42] Matteo Tomei, Marcella Cornaa, Lorenzo Baraldi, and Rita Cucchiara. Art2real: Unfolding the reality of artworks via semantically-awant IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5849-5859, 2019.
mage-tt-imaag translation. In Proceedings of the
[43] Jasper Uijlings, Stefan Popov, and Vittorio Ferrari. Revisiting knowledge transfer for training object class detectors. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1101-1110, 2018.
[44] Thomas Winters, Giuseppe Marra, Robin Manhaeve, and Lue De Raedt. Deepstochlog: Neural stochastic logic programming. arXiv preprint arXiv:2 0t2/1.6..2550 /2574, 2021.
[45] Yao Xue. Nilanjan Ray, Judith Hugh, and Gilbert Bigras. Cell counting by regression using convolutional neural network. In European Conference on Computer Vision, pages 274-290. Springer, 2016.
12

---

Published as a conference paper at ICLR 2023
[46] Yuanyi Zhong, Jianfeng Wang, Jian Peng, and Lei Zhang. Boosting weakly supervised object detection with progressive knowledge transfer. In European conference on computer vision, pages 615-631. Springer, 2020.
[47] Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. Learning deep features for discriminative localization. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2921-2929, 2016.
[48] Xinge Zhu, Jiangmiao Pang, Ceyuan Yang, Jianping Shi, and Dahua Lin. Adapting object detectors via selective cross-domain alignment. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 687-696, 2019.
13

---

Published as a conference paper at ICLR 2023
A TRAINING DETAILS
For the hyper- pperr-parameters the idea was to stay as close as possible to the defaults of the pre-trained standard models although some lightweight tuning was done. Table summary is given for the hyper-parameters used for the different models.
epochs
br_tep__iie
Ingaamma
momentum
weight decay
optimizer
DETR pre-train (retrain)
CLEVE
max 100
0.00X11
(7--)
0.1
8
0.0001
Adamm
DETR pre-train (retrain)
Mols.
0.0001
20 (20)
0.1
0.0001
AdamW
pre-traii train (retruin)
15-20 (20)
RCNN pre-train (retrain)
datasets
max 30
0.005
(5)
0.1
0.9
0.000S
SGD)
RCNN Finctune
all datascts
20
0.001
Adam
DETR Finetune
MNIST
max 20
0.01
Adam
DETR Finctune"
ELEVR/Mols
0,002
20
0.1
8
0,0001
Adamm
Finetune*
CLEVR
RCNN Finetune"
Mols
max 20
0.00001
Adam
DETR masked box loss
CLEVR/Mols
100
0,00XI
0.1
8
0,0001
Adamm
models
Table 4: Overview of hyperparameters for the different models, most hyperparamaters are left default from standard models. Tuning was mostly done on learning rate and learning rate scheduling. For every fold/dataset the best ptoch//r/lr_ste__sa model is used based on validation data.
B DATASETS
We evaluate approach three different datasets: (1) CLEVR-mini dataset, (2) Molecules data set with images of chemical compounds, and (3) an MNIST-based object detection dataset. For each dataset, three subsets, corresponding different domains, used: (1) source domain, (2) target domain, and (3) an out-of-distribution domain (OOD). Source and target domains are split into 5 folds of train and validation sets and an independent test set. Sizes of the different splits per dataset are summarized in Table
14

---

Published as a conference paper at ICLR 2023
Dutaset
Type
Spli
Size (numbeer of samples)
MNIST object detection
Source
train
700
MNIST object detection
Source
validation
300
MNIST object detection
Source
test
1000
MNIST object detection
Target
train
700
MNIST object detection
Target
validation
300
MNIST object detection
Target
test
1000
MNIST object detection
OOD
test
1000
Molecules
Source
train
1400
Molecules
Source
validation
600
Molecules
Source
1000
Molecules
Target
train
1400
Molecules
Target
validation
600
Molecules
Target
test
1000
Molecules
OOD
test
1000
Table 5: Dataset sizes for the different splits. For train and validations splits 5 folds are used.
B.0.1 CLEVR-MINI DATASET
The CLEVR-mini dataset for our experiments is a selection of samples from the CLEVR dataset (21] The different types available in the CLEVR dataset are combinations of shapes (cube, sphere, and cylinder), materials (metal and rubber), and sizes (large and small). Colors are ignored as the images are first converted to grayscale before feeding them the models. For the richly annotated source domain, we randomly select images with only sphere or cylinder-shaped objects (no cubes) and with a maximum of four objects per image and minimum of three objects. For the weakly annotated target domain experiment with two type of annotations. Firstly experiment when we have the class counts of objects in the image available. Secondly, instead of the exact counts of classes in the image the annotations only specify if there is exactly one object class in the image multiple. The advantage of this kind of labeling is that the annotator does not need to count the objects and instead just make a distinction of only one object class image or multiple. The images in the target domain can contain all combinations of object types (including cube-shaped objects) and allow a minimum of five objects per image and maximum of six objects per image. For the OOD dataset also select images with all possible combinations of object types, always with 10 objects per image. Some example images from the CLEVR- mini dataset can be found in Figure
B.0.2 MOLECULES DATASET
The Molecules dataset contains images depicting chemical compounds. For the richly annotated source domain, a procedure similar as described in Oldenhof et al. [29 30 was executed using an RDKit 2 fork for generating the bounding box labels for the individual atoms present the images. In the source domain, we allow the following atom types: carbon (C), hydrogen (H), oxygen (O), and nitrogen (N). In the weakly annotated target domain, we only have the counts of the atoms present which translates to the chemical formula of the molecule in the image (e.g Og)) The same classes from the source domain (C, H, O, and N) are also present in the target domain as well as an extra atom type: sulfur (S). The OOD test dataset consists of 1000 images from the external UoB dataset (36) where chemical compounds containing only the atom types present in the target domain (C, H. O, N, and S). Some example images from the Molecules dataset are visualized in Figure 5
B.0.3 MNIST OBJECT DETECTION DATASET
The MNIST object detection dataset is generated [ using the original MNIST dataset 133 Each image consists of three MNIST digits randomly positioned in the image. The MNIST object detection dataset allows experimenting with a more arbitrary type of weak supervision. Each object this dataset represents digit that can be aggregated. This allows to label an image with only the sum of all digits in the image instead of the class counts of the objects. For the richly annotated source domain digits 7, 8, and 9 are left out. The weakly annotated target domain has all possible digit classes (0-9). The labels of the target domaín only contain the sum of all digits. For the OOD test dataset, images used that contain maximum of four MNIST digits, instead of three digits as in the other domains. Some example images from the MNIST object detection dataset are visualized in Figure 6
15

---

Published as a conference paper at ICLR 2023
Source Domain
Target Domain
Prods from pretrained (on S. dom.)
Preds from pretrained (on S. dom.)
Preds after ProbK'T
G
IN
C
[C]
(C]
[c]
[C]
atom types
{C, N,
atom types
{C, H. N,
{ boxes: (labels, positions))
Y query;= gf
(2= 7C, 1N})
Figure 5; Weakly supervised knowledge transfer with probabilistic logical reasoning (ProbKT), On the left we have source domain where model can be trained using bounding box informa- con((abels.oosition. but only on a limited set of atom types (C,H, O, N). In the middle we can see that the pre-trained model is not able to recognize the sulfur (S) from target domain correctly. On the right we see that the model is able to adapt to target domain after probabilistic reasoning using weak labels (e.g. counts of objects on image) and able to recognize the sulfur (S).
Source Domain
Target Domain
Preds from pretrained (on S. dom.)
Preds from pretrained S. dom.)
Preds after ProbKT
(1)
(3)
2
8
(4)
[5]
4
4
(6
|R]
digits (0, 2, 5, 6)
digits = (0. 1, 5, 8.
Y=
boxes
(lahels, positions))
Y= query:
(sum ofall digits = 20)
Figure 6: Weakly supervised knowledge transfer with probabilistic logical reasoning (ProbKT). On the left we have source domain where model can be trained using bounding box informa- sio((abels.posiitoon) but only on limited set of digits (0, 1, 2, 4, 5, 6). In the middle can see that the pre-trained model is not able to recognize the digit eight (8) from target domain correctly. On the right we see that the model is able adapt to target domain after probabilistic reasoning using weak labels (e.g. sum of digits on image) and is able to recognize the digit eight (8).
C PROBKT AND PROBKT* SUPPLEMENTARY DETAILS
C.I FILTERING SAMPLES
The computation complexity of inference the probabilistic programming module grows with the number of possible worlds. In turn, the number of possible worlds grows with the number of probabilistic facts ń.
One avenue to reduce the computational cost of the inference step is then to artificially reduce the number of probabilistic facts in each image. Let {py,. n = 1,..., fi) and q the corresponding inference query. We compute the filtered of probabilistic facts Py, by setting
16

---

Published as a conference paper at ICLR 2023
if if 2 if s.t. P 2 8
py
and Py
un otherwise.
The parameter [0, 1] is a threshold at which we consider the probabilistic fact as certain. A probability of or 0 effectively discards the probabilistic fact Py.n from the inference procedure. However, we also have to update the inference query to reflect this filtration. We write & the filtered query q.
Example To illustrate this filtration strategy let's consider an MNIST image with 3 digits in the
image: {3, 7). The query corresponds to the class labels in the images. That is q= {3, 7). The object detection backbones outputs 3 box features with corresponding probabilities {py.0, Py.1 P,,,, J. Now let e.g. P3. = 0.99. We can filter out py,1 (i.e the prediction for a digit certain), and compute the filtered query q = {4,7}.
Remark Equation suggests filtering based the output probabilities only. However, also use information about the query for the filtration. For instance, one would only filter out a probabilistic fact if is consistent with the query q- In the example above, it would be wiser not to filter out e.g. P = 0.99 as no images are supposedly present in the image. One should then ideally propagate this probabilistic fact the inference module such to update the weights of the backbone and learn from this error.
C.2 GRADIENT OF THE LIKELIHOOD
The ProbKT likelihood has the following form:
Pr(1) III Pij aEE,
where is "possible world" matrix of indicator variables:
Oaj {
object is of class
otherwise,
and Eg is the set of all possible worlds compatible with the logical annotation
Lemma 1. The gradient of the likelihood has the following form:
OPp(g)
Öpis Cijy
ΣΣ
00
00
where the weight has the form:
Cj P(E(O, =
Σ acElo,=j &
IIII
In of the Hungarian matching the most probable possible word is selected, which corresponds setting the conditional probability P(E = .)) to if object i is paired with label and 0 otherwise. The ProbKT gradient can be interpreted probability weighted extension of the gradient resulting from the Hungarian matching.
D FULL RESULTS
In Table 6, we present the full results for the MNIST experiment. We report the count accuracy (i.e.. correct identification of the digits in the image), accuracy (i.e. correct estimation of the sum of
17

---

Published as a conference paper at ICLR 2023
Model
Type
mnist count acc.
mnist sum
maist mAP (mAP@loU=0.5)
Resnet50-CAM --ect50-CMMMM-- (baseline)
In-distribution
0.044 + 0.041
0.506 0.063
0.003 + 0.003
Besnet50-CAM (baseline)
OOD
0.01 0.009
0,015 1 0.004
0.003 = 0.002
Resnet50-CAM CAM (baseline)
Source Domain
0.127 1. 0.132
0.649 1 0.108
0.005 1 0.004(00008
DETR (Pre-trained)
In-distribution
0.26 + 0.012
0.262 = 0.001
0.518 = 0.014 (0.667
DETR (Pre-trained)
OOD
0.173 1 0.01
0.177 + 0.009
0.51 1: 0.012 00.6322200000
DETR (Pre-trained)
Source Domain
0.859 + 0.031
0.86 + 0.031
0.781 + (0.957 0.008)
DETR (ProbKT)
In-distributin
0.662 1 0.064
0.664 0.065
0.615 1. 0.025 (0.856 0.037)
DETR (ProhKT)
OOD
0.532 + 0.083
0.533 + 0.082
0.591 + 0.03 (0.845 0.038)
DETR (ProbKT)
source domain
0.878 + 0.023
0.879 = 0.023
0.737 = 0.014 (U.952 0.009)
RCNN (Pre-trained)
In-distribution
0.292 + 0.005
0.298 + 0.005
0.632 + 0.014 ( (0.855 0.002)
RCNN (Pre-trained)
OOD
0.205 + 0.004
0.212 = 0.004
0.631 + 0.013 ( (0.683 + 0.002)
RCNN (Pre-trained)
source domain
0.961 1 0.008
0,961 1 0.008
0.917 1 0.021 (0.988 1 0.002)
RCNN (ProbKT)
In-distribution
0.902 + 0,005
0,903 = 0.005
U.786 + 0,U21 (U.974 + 0.001)
RCNN (ProbKT)
OOD
0.863 +: 0.008
0.865 + 0.008
0.778 + 0.021 (0.97 +: 0.001)
RCNN (ProbK'T)
source domain
0.967 + 0.004
0.967 = 0.004
0.873 + 0.016 (0.989 = 0.001)
Table 6: Results of the SUM experiments on the MNIST object detection dataset. Reported test accuracies over the folds.
the digits in the image) and the mean average precision (mAP) (i.e. a common object detection metric that reflects the ability to predict the positions and labels of the objects). We observe that the Resnet baseline performs poorly, lacking the necessary logic to process this dataset. We used both DETR and RCNN as object detection backbones in our experiments, showing high test accuracies when fine-tuned with our approach. As the results suggest, RCNN backbones lead to better performance than the DETR backbone.
Model
Data Domain
CLEVR count acc.
CLEVR mAP (mAP&1ol)-0.55
Mol. count, nze
Mol. mAP (mAPG))OL--0.5)
RAsM
target domain
0.97 0.005
0.036 0.014 (0.200 0.071)
0.978 0.004
0.0 0.0 + D) Iuaa
ResnetO0-CAM
OOD
0.831 0.016
0.024:: 0.010 (0:153 (:03::00.044
0. 0.0
Restet50-CAM
source domain
0.953 0.009
0.035 + 0.019 (0:178 + 0.084)
0.828 0.021
0.0 0.0 +)
WSOD-iransfer
target domain
0.944 0,004
0.804 0.065 (0.968 1. 0.001)
0,001 0.0
0.018 0.00L (0.061 0.011)
WSOD-transfe
0.73 0.011
0.79 0.005 (0.900 0.001)
0.003
I/a"
WSOD-transfer
source domain
0.989 = 0.001
0.926 0.001 (0.955 0.00
0.0 4: 0.0
0.021 0.003 (0.069 0.009)
DETR. joint
target domain
0.159 0.133
0.579 d: 0.012 (0.084 :: 0.019)
0.357 à: 196
0.197 ndal
0.055 (0.481 1 (.071)
DETR. joint
OOD
0.084 0.009
0.534 0.012 (0.06 0.012)
0.024 0.021
DETR.joim
sounce dom.
0.923 0.049
0.908 0.017 (0:052 0.001)
0.232 0.127
0.063 (0.585 0.08)
DETR (Pre-trained)
target domain
0.0
0.498 0.015
0.024)
0.464 0.083
0.314 0.006 00.542 0.006)
DETR (Pre-traimol)
OOD
0.0 0.0
0.477 0.013 (0.531 0.021
0.002 0.001
naa
DETR (Pre trainel)
sourcr domain
0.97 : 0.009
0.905 2: 0.009 (0.992 1: 0.001)
0.581 1 0.022
0.108 1: 0.005 (0. 7272222222 0.001)
(DETR)
target domain
0.949 0.005
0.728 + 0.014 (0.99 0.003)
0.589 0.042
0.373
0.02 (0. (0.689 0.045)
(DFTR)
OOD
0.741 0.038
0.606 0.017 (0:977 0.004)
0.008 0.008
(DFTR)
source domain
0.985 0.004
0.937 + 0.006 (0.003
0.001)
01227 + 0.066
0.371 0.021 (0.649 0.041)
PrbKT((EER)
target domain
0.946 - 0:014
0.803 + 0.011 (0.080 0.006)
01508 0.027
0.204 + 0.02 (0.507 0.0144
ProKKTDDERR
OOD
0.715 0.006
(006))
0.004
ProKTTDDER)
souce domain
0.987 0.003
0.948 0.005 (0.005
0.001)
0.349 0.025
0.38 0.013 (0-713 0.006)
(pre
target domain
0.014 (0.598 0.013)
0.592 0.007
0.568
0.005 (0.785 0.004)
RCNN (pee-trained)
OOD
0.0 0.0
0.582 0.012 (0.003 0.011)
0.348 +: 0.006
nía'
RCNN (pee crained)
source domain
0.988 = 0.002
0.984 4: 0.01 (0.996 + 0.0)
0.968 1: 0.004
0.737
0.005 (0.979 dt: 0.0)
(RCCK)"
target domain
0.974 0.004
0.855 0.025 (0. 09.994555 +:: 0.001)
0.345 0.006
0.042 uaal
(0.788 0.073)
(RCKKT(RCNN)
OOD
0.901 0.017
0.827 0.022 (0.991 0.001)
0.592 0.032
(RCNN)
souree domain
0.993 = 0.002
0.95 1: 0.021 (0.998 + 0.0)
0.96 2: 0.003
0.665
0.01 (0.974 1: 0.004)
ProcKT(CCNN)
domain
0.975
0.003
0.856 0.039 (0.993 1: 0.001)
0.942 0.009
0.289
0.041 (0.829 + 0.054)
ProkKTRRCNN)
OOD
0.69
0.042 (0.991 0.001)
0.003
n/a"
PobKT(RCNN)
source domain
0.995 d: 0.002
0.941 + 0.041 (0.008 0.001)
0.96 = 0.002
0.666
0.005 (0.979 0.002)
Table 7: Results of the experiments for the datasets: CLEVR-mini and Molecules. Reported test accuracies over the 5 folds. Best method is in bold for each metric and data distribution.
OOD test set of Molecules datasct has no bounding box labels.
18

---

Published as a conference paper at ICLR 2023
E SOURCE CODE AND DATASETS
The source code and basic instructions are available on https://githubb: com/ mo Iden/ ProbKT The source code integrates features from the Weights Biases (WandB) platform [5]. Basic features are supported without the need for an account on WandB but to make full use of all features recommend create account.
Datasets can be downloaded here:
CLEVR-mini dataset http::/ttsssse....tccc./o//////a
Molecules dataset httt:p/ee/Eigggerrr..mmm////////
MNIST
object
detection
dataset
https://llggarre.comm//
c760cee0260005244333
ProbLog script used in the ProbKT Probabilistic logical reasoning framework for counting of objects on an image (as on CLEVR-mini dataset):
use_module (library (lists)) nn (mnist_net, [x], Y, 10,1, 4, 6, 8, 9,10, :: digit (x, Y)
count ([),X, 0). count [[X] (i,TTlXX,,,, :- count 2,, Y is 1+2. count ([X1 2,,X,20 2) x1\=X, count (T, X, 2)
countall (List, C) :- sort (List, Listl), member (X, Listl, count (List, X, C).
(I], ([HIT] A) roll (T, L)) digit (H,
countpart (List, [),
countpart (List, [H]T), (FIL)) :- countall (List, F),, countpart (List,
count_objeets (X, L) :- roll (x, [1, Result), countpart (Result,
The query q in the case of class counts would be count_objects (X, L, C). For example an image X with small metal cube and 3 large rubber cylinders would result in the following query: count_object s (X,
(1,3) [1,]))
ProbLog script used in the ProbKT Probabilistic logical reasoning framework for aggregating the digits on an image:
usd_me (library (lists))
nn (mnist_net, [X], Y, [0, 1, 3, 5,5, 6, 6,7,, digit (X, Y).
sum 0), sum ( [X T], :- sum (T, y is X+2.
roll (1), L)) roll ([HIT), A, roll (I, [XIA], digit (H, Y)
sum_digits (x, :- roll (x, [], Result), sum (Result,
The query g in the case of sum of digits would be sum aun_digits (X, Y). For example an image X with as sum of digits 12 would result in the following query: sum cim_digits (X, 12).
ProbLog seript used the ProbKT Probabilistic logical reasoning framework for taking into account non-exact counts images
19

---

Published as a conference paper at ICLR 2023
uis_module (library (lists))
on (mnist_net, [X], Y, [0, 1, 2, 3, 5, 6, 1,8,,,11111)) :: digit (X, Y)
count ([],X, 0)
count ([X]T], xX) :- count xT,x, Y is 1+2. count ([X1 T] x, 2) xI)-X, count (T, 2)
countall X,,st, A=0,
sort (List, Listl), member (x, Listi), count (List, X,C), C). countall (List, A) :- A-1,
sort (List,
member (x, Listl)
count (List, x, R),
R>C.
countall (List, C, A=-1,
sort (List, Listl), member (x, Listl), count (List, R),
R<C.
(oll L) L), roll A, :- roll (T, [XIA],L), digit (H,
countpart (List, [], [], [])
countpart (List, []] [FIL], [A]B]]: :- countall (List, F, countpart U,
range_coure countpart (Result,,,CC. C,
(x, L, S) :- roll (X, Result),
The query in the case of non exact counts of objects would be range_counts ccoun__objecte (x, S). For example image X with exactly one metal small cube and multiple rubber large spheres would result in the following query: range_ lanng__count_objcct (X, ss_metal_ccubbl cube, rebber___ubbb_peeeeee_e_
[1,1], [0,1]).
E.1 INFERENCE EXAMPLE FOR MNIST DATASET
To illustrate the inference process let us follow the evaluation of the clause sum ( [x11, x2] 8), what can result from query sum_digi digits (x, 8) in case of two visible digit the image
This clause is true if and only if X, + X₂ 8.
In case of MNIST digits (f0, 1,..., 9}) enumerating the possible worlds would give the following set:
{0, (2,), (8,
After summing the probability of all possible worlds we get:
pi (0)p2 (8) + pı (1)) - ... + pi (0))228),
(4)
where pr and Pa are the distribution of random variable X, and X₂ respectively. Or in general form:
(V) = Σ
-X))
(5)
X1
20

---

Published as a conference paper at ICLR 2023
As expected the distribution of the sum is the convolution of the distributions of the two terms. This observation trivially generalizes to more than two terms. The cost function corresponding to the maximum likelihood estimation is the negative log- reeliikel - lo (Y)).
21


Reviews:
[{"review_id": "g-uKgOPgBtf", "reviewer": "Reviewer_tNrs", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper presents a new way of doing object detection with weak supervisions. The model proposed is called ProbKT and it allows any object detection model to be trained on a general and rich source domain, and then transfer the acquired knowledge on a weakly annotated target domain. The novelty of this work lies in the fact that it allows to train object detection models with arbitrary types of weak supervisions. ", "summary_of_the_review": "**TL;DR:** the paper seems novel and relevant. however, its relationship with deepproblog needs to be clarified ", "strength_and_weaknesses": "**Strengths:**\n\n1) The paper addresses the important problem of training object detection models with weak supervisions\n2) It is the first paper to allow for more complex weak supervisions\n3) The experimental analysis is comprehensive and well documented\n\n**Weaknesses:**\n\n1) The authors seem to be sweeping under the rug their reliance on DeepProbLog. Indeed, if I understood correctly, the reasoning and backprogation is done through the DeepProbLog framework. If this indeed the case then it needs to be made clearer and a short introduction on the framework should be given. \n2) It is not clear the exact expressivity admitted by the weak supervision, is it only limited to rules? Is it full first order logic?\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "_Evfbskehz", "reviewer": "Reviewer_ijtT", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Detailed annotations are often not available in practice, and only high-level image information is available on the target dataset; this work presents a fine-tuning method using ProbKT to update object detection models from a pre-trained architecture. Nevertheless, the presented knowledge transfer method can be built upon arbitrary types of weak supervision on the target domain, such as complex logic statements. The authors empirically show on three types of datasets that fine-tuning with ProbKT leads to significant improvement in the target domain and gives better generalization compared to existing baselines.", "summary_of_the_review": "Overall, the presented idea is novel and helps equip the model with logical reasoning as humans on arbitrary input. The experiments show its effectiveness, especially I can imagine it can help more when the quantity of the dataset is not large enough to train a robust model.", "strength_and_weaknesses": "Strength:\n1. The introduced method for weakly supervised knowledge transfer learning is quite interesting. \n2. The introduced method allows arbitrary types of supervision on the target domain can greatly alleviate the annotation burden.\n3. Extensive experiments show the proposed method is quite effective.\n\nWeakness:\n1. The idea has been published before, and I do not see improvement or substantial differences over the published one.\n\n2. The experiments are conducted on simpler content datasets, like MNIST. I expect some explanation of how to use the ProbKT for large-scale and complex content datasets on object detection, ex: what will the queries look like? \n[ Authors provided feedback on this question, for instance, they said the model could be supervised on the count of detected objects, etc., on the COCO dataset. However, this counting information is already provided from the ordinary ground truth. Specifically, the number of bounding boxes for objects already provided the model with object counts. Therefore, when you supervised the model with queries like, “there are at least 5 dogs in this image” or “there are more humans than animals in this image,” what additional information did we provide to the model? In other words, I am still not fully convinced of the author's answer.]\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Nj6jzWNm5f", "reviewer": "Reviewer_q7Gr", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces a new method for weakly supervised object detection with knowledge transfer, termed ProbKT. The basic idea is formulating the weakly supervised detection problem as the probabilistic logical reasoning so that the detector can be trained with arbitrary types of weak supervision.  The output of the detector is fed into a neural probabilistic logical reasoning module. Several modifications are also proposed to reduce the computational cost of probabilistic programing specifically for object detection. Experiments show that ProbKT outperforms WSOD-transfer [41] and several baselines on CLEVR-mini, Molecules, MNIST on both detection and counting tasks.", "summary_of_the_review": "Based on the novelty, the clarity and the thorough experiments, this paper is good for acceptance at ICLR.", "strength_and_weaknesses": "# Strengths\n+ By formulating the weakly supervised object detection as the probabilistic logical reasoning, a unified detection method can be trained on dataset with different supervision signals, which is a novel approach for weakly supervised knowledge transfer.\n+ The paper presentation is clear, with well-defined terms and concepts.\n+ Ablation studies are thorough. The effects of proposed components are validated on several datasets.\n# Weaknesses\n- It would be great if the method can be evaluated on datasets with real world objects like COCO or LVIS.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "CiF_mfQG_lt", "reviewer": "Reviewer_Mz4p", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a framework based on probabilistic logical reasoning that allows to train object detection models with arbitrary types of weak supervision. The proposed architecture consists of two components: symbolic reasoning and deep learning architecture. \nSuch a structure is easy to implement weakly supervised learning in the target domain. Experimental results are given to demonstrate the good performance of the proposed model.  \nThe main contribution of the paper is the incorporation of knowledge from other domains into object recognition framework. It is beneficial to use the proposed ProbKT to improve recognition performance on target domain and better generalization compared to existing baselines. \n\n", "summary_of_the_review": "The use of probabilistic logical reasoning for improving the performance of object detection is interesting topic in the AI society.  The integration of probabilistic logical reasoning and deep learning model results in performance gain for object detection.  ", "strength_and_weaknesses": "The paper proposes a new framework that incorporates probabilistic logical reasoning method into the object detection models. \nStrength:  Such a structure is easy to implement prior knowledge from other domains into deep learning models, is also able to train object detection models by leveraging richly annotated datasets from other domains and allowing arbitrary types of weak supervision on the target domain. \nWeakness: \nAlthough the proposed architecture integrates symbolic reasoning into deep learning architecture, it is still unclear from theoretical point of view that why the relabeling will provide performance improvement during symbolic reasoning. \nWith regard to \"weakly supervises learning\", the paper provides a simple example \"The sum of all digitals in the image is considered as weak supervision\". How does the cost function is implemented in the probabilistic logical reasoning stage\nThe proposed model includes the probabilistic reasoning procedure, which is of higher computational complexity. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster