Hybrid Neuro-Symbolic Reasoning based on Multimodal Fusion
Authors: Subrata Das
Bodong Zhou
Venue: Submitted to ICLR 2023 (2023)
Keywords: Neural Networks
Deep Learning
Symbolic Reasoning
Multimodal Fusion
Word Embedding
Rule-based Reasoning

Abstract: Deep neural models and symbolic Artificial Intelligence (AI) systems have contrasting advantages and disadvantages. Neural models can be trained from raw, incomplete and noisy data to obtain abstraction of features at various levels, but their uninterpretability is well-known. On the other hand, the traditional rule-based symbolic reasoning encodes domain knowledge, but its failure is often attributed to the acquisition bottleneck. We propose to build a hybrid learning and reasoning system which is based on multimodal fusion approach that brings together
advantageous features from both the paradigms. Specifically, we enhance convolutional neural networks (CNNs) with the structured information of ‘if-then’ symbolic logic rules obtained via word embeddings corresponding to propositional symbols and terms. With many dozens of intuitive rules relating the type of a scene with its typical constituent objects, we are able to achieve significant improvement over the base CNN-based classification. Our approach is extendible to handle first-order logical syntax for rules and other deep learning models.

Under review as a conference paper at ICLR 2023
HYBRID NEURO-SYMBOLIN REASONING BASED ON MULTIMODAL FUSION
Anonymous authors Paper under double-blind review
ABSTRACT
Deep neural models and symbolic Artificial Intelligence (AI) systems have con- trasting advantages and disadvantages. Neural models can be trained from raw. incomplete and noisy data to obtain abstraction of features at various levels, but their uninterpretability is well-known.
the other hand, the traditional rule- based symbolic reasoning encodes domain knowledge, but its failure is often at- tributed to the acquisition bottleneck. We propose to build a hybrid learning and reasoning system which is based multimodal fusion approach that brings to- gether advantageous features from both the paradigms. Specifically, we enhance convolutional neural networks (CNNs) with the structured information of 'if-then' symbolic logic rules obtained via word embeddings corresponding to proposi- tional symbols and terms. With many dozens of intuitive rules relating the type of scene with its typical constituent objects, we are able to achieve significant improvement over the base CNN-based classification. Our approach extendible to handle first-order logical syntax for rules and other deep learning models.
INTRODUCTION
Deep learning technology is being employed with increasing frequency in recent years LeCun et al. (2015) Schmidhuber (2014). Various deep learning models have achieved remarkable results in com- puter vision Krizhevsky et al. (2017), remote sensing Zhu et al. (2017), target classification in SAR images Chen al. (2016). and speech recognition Graves et al. (2013 Hinton et al.) (2012). In the domain of natural language processing (NLP), deep learning methods are used to learn word vector representations through neural language models Mikolov et al. (2013) and performing composition over the learned word-vectors for classification Collobert et al. (20TT). Convolutional neural net- works (CNNs), for example, utilize layers with convolving filters that are applied to local features. CNN is widely used for image tasks and is currently state-of-the-art for object recognition and de- tection. Originally invented for computer vision, CNN models have subsequently been shown to be effective for NLP and have achieved excellent results in semantic parsing Yih et al. (2013), search query retrieval Shen et al. (2014), sentence modelling Kalchbrenner et al. (2014), and other tradi- tional NLP tasks Collobert et al. (001T.
However, the success of deep learning comes at a cost. The first and foremost its reliance on large amounts of labeled data, which often difficult collect and entail slow learning process. Second, deep models are brittle in the sense that trained network that performs well on one task often performs very poorly on new task, even if the new task is very similar the one was originally trained on. Third, they are strictly reactive, meaning that they do not use high-level processes such as planning, causal reasoning, cannot be used which can often reduce the burden of acquiring training data which is often expensive to collect. Purely data-driven learning can lead to uninterpretable and sometimes counter-intuitive results Nguyen et al. 2014 Szegedy et al. (2013).
analogical reasoning. Fourth, human expertise
The sub-symbolic neural approaches allow us to mimic human cognitive thought processes by ex- tracting features at various levels of abstraction from direct observation and thereby facilitate learn- ing. But humans also learn from general high-level knowledge expressed declaratively in logical syntax. A representation language allows recursive structures to be easily represented and manipu- lated, which is usually difficult in neural learning environment. But a symbolic reasoning system not good to adapt environments by learning and reasoning based on traditional theorem-

---

Under review as a conference paper at ICLR 2023
proving, which can be computationally expensive. Moreover, a purely symbolic system based on traditional Al requires enormous human effort as knowledge are manually programmed and not leamned Central to classical AI is the use of language-like propositional representations to encode knowledge. The symbolic elements of a representation in classical Al - the constants, functions, and predicates are typically hand-crafted. Inductive Logic Programming MMuggleton (1990) methods learn hypothesis rules given background knowledge, and a set of positive and negative examples. The systems we have discussed until now do not model uncertainty which is essential in practical applications. Various probabilistic logics Halpern (2005) and Markov Logic Networks Richardson & Domingos] (2006) (MLNs) handle uncertainty using weight attached every rule. Practical ap- plications of these networks have been limited as inference is not scalable to large number of rules.
It is, therefore, desirable to develop a hybrid approach, embedding declarative representation of knowledge, such as domain and commonsense knowledge, within a neural system. this paper, hybrid approach is applied to indoor scene classification, which has been extensively studied in field of computer vision Chen et al. (2018). However, compared with outdoor scene classification, this is an arduous issue due the large variety of density of objects within typical scene. In addition, high-accuracy models already exist for outdoor scene classification while indoor scene classification not. In order accomplish our objective, the acquisition, representation, and utilization of visual commonsense knowledge represents a set of critical opportunities in advancing computer vision past the stage where it simply classifies or identifies which objects occur in imagery Davis al. (0115)
The contributions of this paper is summarized as followed:
A joint representation multimodal fusion framework is applied to exploit the early fusion of vectorized logical knowledge and images for the task of indoor scene classification. Experiments show that higher classification accuracy is obtained compared to traditional image classification methods.
A 'if-then' logical knowledge system is built based on reviews of each indoor scene class which are scraped from Google open source, through Word2Vec and BERT embedding. This helps to get a better contextual representation of words detected by object detection.
A unique rules embedding approach is proposed, which allows to converge 'if-then' logic of probability with image representation. The embedding approach has different represen- tations during training and inference process.
The rest of the paper is organized as follows. The next section 2 surveys the related work. The hybrid framework is explained in section 3 Section details implementation and evaluation of experiments. Finally, we conclude with some future directions in 5
2 RELATED WORK
Hybrid neural-symbolic systems concern Chen et al. (2016) Garcez al. (009 Hammer & Hit- zler 2007 Rosenbloom et al. (017 Sun (1994 Wermter & Sun (2001) the use of problem-specific symbolic knowledge within the neurocomputing paradigm, specifically, symbolic domain and com- monsense knowledge within the deep learning paradigm in our case. They are useful for enhancing various tasks, including logical inferencing, extracting relational knowledge Guillame- et al. 2210 Gust et al. (2007), image classification, and action selection.
Combination of logic rules and neural networks has been considered to construct network architec- tures from given rules to perform reasoning and knowledge acquisition. Neural-symbolic systems, such as EBL--NN Shavlii & Towell 1989), KBANN KBANNSZzegdyy et al. (2013) and Carcez al.) 2009 LENSR Xie et al. (2019). like proposal, deal with propositional formulae. KBANN, for examplc, maps problem-specifee domain thcorics, represented in propositional logic, into neural net- Propositional symbols
works and then refines this reformulated knowledge using back-propagatie are directly represented as nodes whereas we vectorize each propositional symbol as its semantic representation and appended to the abstraction of low-level observations.
Other neural-symbolic systems are exploring on knowledge graph Chen et al. (2020 Kampffmeyer et al. Li099Li et al. Zablocck et al. (2019) which is natural symbol. It is not only
2

---

Under review as a conference paper at ICLR 2023
mantic network describe entity relationships, but also formal description framework for general semantic knowledge.
A large amount of neural-symbolic approaches focus first-order inference Zhang et al. (2020 Marra et al. (2020) Yang & Song (2020 Cai et al. (202T). But some do not allow one to learn vector representations of symbols from training facts of knowledge base, such as SHRUTI Shastrii (1999), Neural Prolog Ding et al. (199)), CLIP++JFranca et al. (2014) and Lifted Relational Neural Networks Sourek et al. (2015). Neural Reasoner Peng al. (2015) translates query representations in vector space without rule representations and can, thus, not incorporate domain specific knowl- edge. The Neural Theorem Prover Rocktäschel & Riedel (2166) and Unification Neural Networks (2011) build upon differentiable backward chaining, but the for- mer operates on vector representations symbols whereas the latter on scalar values. Grounded Abductive Learning (GABL) Cai et al. (2021) is proposed to enhance machine learning models with abductive reasoning in a ground domain knowledge base, which offers inexact supervision through set of logic propositions.
Different frameworks of neural-symbolic system rely on various logics Raedt et al. (011 For example, Logic Tensor Networks Donadello et al. (2017) is based on first-order logic, Lifted Rule Injection/Demeestee et al. (2016) exploits implication rules, Semantic Loss Function Xu et al. (2018) focuses propositional logic. Other deep learning systems such as TensorLog Cohen (2016) uses datalog, while DeepProbl Mag et al.] (2019) uses clausal logic. Among these frameworks, Semantic Loss Function and DeepProbLogg not only embed logic, but also add probabilistic neural networks, which are two principal factors of reasoning.
Like our method, DeepProbLog Manhaeve al.. (2019) purposes a framework where expressive probabilistic-logica modeling and reasoning are combined, which could be trained end-to-end. Yang al. Yang Song (2020) proposed Neural Logic Inductive Learning (NLIL), an efficient differentiable ILP framework that learns first-order logic rules to explain problem in the scope of inductive logic programming (ILP). Marra al. |Marra et al. (2020) presented Relational Neural Machines (RNM), novel framework to converge deep architectures and probabilistic logic reason- ing. able to recover both classical learning case of pure sub-symbolic learning, and Markov Logic Networks in case of pure symbolic reasoning. Mao et al. [Ma et al. (2019) proposed the neuro-symbolic concept learner (NS-CL), which able to represent object-based scene and trans- late sentences into executable, symbolic programs.
3
HYBRID NEURO-SY METHOD
3.1 OVERVIEW
Logical knowledge representation is symbolic in nature, i.c. the data structures under consideration basically consist of words over some language. A logic program [loyd (1984) is set of (univer- sally quantified) disjunctions, called clauses or rules, which turn consist of atoms and negated atoms only. Definite rules of 'if-then' type have conjunction of atoms as antecedent and one atom as consequent. Successful connectionist architectures, however, can be understood as networks (essentially, directed graphs) of simple computational units, in which activation propagated and combined in certain ways adhering to connectionist principles. In many cases like, for example, in multi-layer perceptrons, the activation is encoded as real number; input and output of such networks consist of tuples (vectors) of real numbers.
In order to integrate logic and connectionism, we, thus, have bridged the gap between the discrete, symbolic setting of logic, and the continuous, real-valued setting of artificial neural networks. In this paper, we propose an approach to multimodal learning which involves relating information from multiple sources. Specifically, we focus on learning representations for images which are coupled with vectorized features of propositional rules, along the line of multimodal video/audio deep learning in Ngiam al. (201I) or multimodal deep Boltzmann machine (DBM) Srivastava & Salakhutdinov (2012). The bimodal DBM in our case models the joint distribution over image and symbolic knowledge inputs. The joint distribution over the multi-modal input variables is written as
3

---

Under review as a conference paper at ICLR 2023
1
p( 00) (is x
Σ eep(( (peplliietted Softmar Symbolic Pathway
+ Gaussian Image Pathwoay
+ Joint Fully Connected Layer)
where all hidden variables with superseripts in parentheses, Z is the normalization constant depending the number of propositional symbols in all the rules. The image-specific DBM uses Gaussian distribution to model the distribution over real-valued image features. Similarly, rule- specific specific DBM uses Replicated Softmax to model the distribution over word count vectors.
Word-embeddings of symbolic and subjective rules adopt divergent representations, whilst Word2 and BERT are introduced in this paper to represent the contextual knowledge. The us- age of BERT representation will provide stronger semantic contextual meaning and relevance with each label, so that the hybrid network is able understand symbolic rules better Katchbrraccrrrrner al. (2014). The Word2 representation captures many linguistic regularities and preserves semantic similarity meaning. We set the value of the absent modality to zero when computing the shared representation, which is consistent with the feature learning phase. Word-embeddings of symbolic rules and objects images are extracted and input into the Word2Vec and BERT model.
The proposed hybrid framework with CNN has been applied to indoor scene classification sce- nario with classes, namely, library, museum, concert, church, mall. Here how we prepare the labelled instances for training and text from image corpus and a set of logical rules, shown in
I In the scenario, for example, for an input image corpus, the pixel representation of each image fed into the hybrid model but coupled with zeros if object is identified in the image, or the vectorized representation of the identified objects within the image; so if the image scene is of type "library" and if the only identified object is "shelf" then an abstraction of the image pixel array at certain level is coupled with the word embedding of "shelf"" The combined representation then be- comes a training sample with the label "library" and propagated into fully connected classification network.
On the other hand, if we have domain propositional logic rule "if shelf & table then library (0.8)", then vectorization of "shelf & table" is coupled with zeros for the image modality. The combined representation also then becomes training sample, but in this case the component corresponding to "libvary" of the 1-to-C coding of the label will have 0.8 instead of 1. This hybrid approach is an early fusion of modalities. Moreover, order to integrate Word2Vec and BERT representations, logic rules are alternated into these two representations to get specific vectorization explained above, then merged horizontally into a vector.
We have made of publicly available images and textual blogs to generate hundreds of domain rules. Like the example above, automatically from applying Bayes' probability and then merged with additional hand coded domain rules. Each image is also optionally tagged with number of objects. We can leverage on one or more of many existing object detection frameworks, such as Spatial Pyramid Pooling (SPP) He et al. (014)) OverFeat Serrmanteeeneteeeeee et al. (2013), Multibox SSD Liu eet al. (2016), Fast R-CNN Girshick (2015). YOLO Redmon et al. (2016), and Faster R-CNN Ren al. (2015), and TensorFlow based Google object detection Caffe tool. The hybrid framework strongly improves over the basic CNN as illustrated by examples in Figure I but much more details are in the implementation and evaluation section. To the best of our knowledge, this is a state-of- the-art approach to tightly integrate traditional 'if-then' logic rules with CNN.
3.2 RULES GENERATION
Two different types of logical rules are generated. One is initiated by object detection algorithm and Bayes' theorem, another is manually preset by us. Our main approach is based automatically generated rules from Bayes' theorem. However, we have compared the effect of two types of rules within fusion network in our experiments, shows the detail.
4

---

Under review as a conference paper at ICLR 2023
Library MuseuumConcorCCuucc Mall
Fully Connected Layers
+
word2vec Transformation
BERT Transformation
CNN High- Level Features
if shelf & table then library (0.8) window & sculpture then church (0.9)
sheff, table, bookcase
Object Detection
Figure 1: Hybrid architecture example
3.2.1 AUTOMATICALLY GENERATED RULES
For each image, through Fast R-CNN object detection algorithm Girshick 2015), we tag them with number of objects anywhere of 10. For example, in "library" image, group of objects, "chair, building, chair, chair, chair, shelf, shelf. table, couch, shelf" is given. These 10 tags may contain repeated objects, we implement the counting of repeatedly additions objects there are in each scenario. These objects which emerge less than 10 times in scenario will be subtracted, since each scenario has 100+ images and the probability is too low be counted which may influence the accuracy of the representation of image rules. Finally we get 2 to 8 objects each image. For example, an image of a "library" is likely to contain "shelf, chair; table, bookcase".
figure
how many
Combination theory helps to form rules subsets from single objects to multiple objects. k objects are selected from a set of n objects to produce subsets without ordering. The number of such subsets k)! where & here is set 2 and 3, which means that the combinations
is denoted by C = n//(( of 2 objects and objects generate and will form rules of features and 3 features. Same as previous subtraction, these combinations which appear less than 10 times avoid redundant feature rules generation. Thus, the probability of single objects or multiple object combinations can be calculated for each scenario, that is, the probability of "shelf", "shelf & table", "shelf & table & bookcase" emerge in a library image. Based on the Bayes' theorem, the probability of certain scenario when object example, we use the theorem to calculate that if the combination of "shelf & bookcase" is in scenario, the probability of "library" 95%. Table shows some image rules calculated using Bayes' theorem.
removed ensure
combination of objects emerge
be calculated. For
3.2.2 HUMAN ENCODED RULES
Within these artificially generated 'if-then' logics, object keywords in the 'if-then' logic are picked randomly by hand and are contextually relevant. The probability of each logic are set as follows: if an object appears alone, haphazardly give the probability of 0.75 or 0.8; if it appears by combination of two or three objects, then we add 0.05 or 0.1 at random to the probability of one or two objects correspondingly. Table 2 shows some logic rules initialized manually.
5

---

Under review as a conference paper at ICLR 2023
Table I: Automatically Generated Image Rules Examples
Prob
Class
Feature
Feature
Feature
80
library
shelf
library
chair
table desk
100
library
shelf
bookcase
museum
picture
100
museum
furniture
sculpture
100
concert
curtain
100
concert window
curtain
chair
100
church bench
74
church window
sculpture
houseplant
100
mall
vehicle
82
mall person
building
plant
3.3 MULTIMODAL NETWORK FRAMEWORK
CNN
WerdEves Vestors
Indoor Boone Catagorics
Obgdda
Ecendiation Laym
Faster
Dart Vectors
Figure 2: Multimodal Network Framework
We have constructed multimodal neural network as shown in figure Firstly, 9-layer CNN model is utilized to extract representation of image features. Note that our main objective is not to come up with best CNN architecture for scene classification but rather to show the relative improvement in the performance when the base architecture is hybridized with symbolic approach.
Next, Faster R-CNN model is leveraged to implement object detections. 10 objects have been ex- tracted from each image. We then generate several permuted 'if-then' rules based on these extracted objects through Bayes statistical method. These logical rules represented by Word2Vee and BERT embeddings respectively. The pre-trained CBOW-based Word2 Vec model is exploited to generate Word2Vec vectors for these rules. What's more, for engendering BERT vectors, the struc- ture of BERT classifier is as follows: the small pre-trained BERT model from Tensorflow is loaded, followed by fully connected layer with 5 units (number of classes) with softmax activation func- tion. After training, extract intermediate vector BERT representations of these knowledge.
After acquiring image features from CNN network, Word2Vec and BERT representations logical knowledge rules, we perform multimodal fusion method them. Specifically, superimpose the dimension size of these image and rule vectors as the input dimension size of our subsequent fusion network. For each dimension of data, besides its vector, set zero values for these posi- tions occupied by other two types data. All the number of training image and probability based training rule entries concatenated for multimodal fusion training. However, when predicting.

---

Under review as a conference paper at ICLR 2023
concatenate the image and its corresponding rules in the image into vector input, which is different from training. Obtaining joint representation of these multimodal features, a multi-layer perceptron is used to do fusion classification.
3.4
MODALITIES EMBEDDING
3.4.1 TRAINING
For training, multiple label inputs for each image formed as follows, which is divided into two parts. One is the image representation and the other is the rules embedding. The overall input dimension is 300+512 12+226.. In terms of image features, the 300-d plus 512-d zero vectors are con- catenated with 256-d last layer of CNN. For rules, the composite 300-d Word2 word-embedding and 512-d BERT word-embedding as is done for image tagging. Then, the vector amplified with a 256-d vector of zeroes, which is used align with the overall dimension to obtain training instance for the rule. The label corre- sponding to this instance is the consequent of the rule. We replicate the rules instance proportional associated confidence probability needed balance two sets of training instances generated from the raw images and rules. For example, if possibility of the rule is 100%, this instance is then replicated 10 times.
formed using respective conditions in the antecedent of rules
3.4.2 INFERENCE
By including the individual word-embeddee of the tagged objects, if any, of the image, the compos- ite 300-d Word2Vec word-embedding and 512-d BERT word-embedding are then augmented with high-level features of 256-d for each labeled image extracted from CNN. If there objects with the image then the 300+512+256 dimensional input vector for the image is formed by padding with zeroes.
tagged
IMPLEMENTATION AND EVALUATION
4.1
DATASET
We have totally collected 733 images for all indoor scene classification scenario, almost equally distributed among the classes "library", "museum". "concerr", "church" and "mall". Among 733 images, there are 440 used for training. 146 for validation and 146 for testing. The dataset is orig- inally provided by MIT, which contains 67 Indoor categories, and total of 15620 images. The number of images varies across categories, but there are at least 100 images per category. These images are of various dimensions and pixel resolution, but we have wrapped each to the fixed di- mension 96x96.
4.2 NETWORK SETUP
In our proof-of-concept demonstration for hybrid learning, the CNN model as in is implemented in TensorFlow. We have made use of Google's pre-trained Word2Vec model via the Python package genism. The dimension of the embedding vector 300. The BERT model is pre-trained Tensorflow small English uncased BERT L-4-H-512 model. The text classifier is built on that which has single neural layer using Softmax activation. The dimension of the embedding vector is 512.
The following parameters are used for training the multimodal hybrid network with rules transfor- mation: Number of epochs 50; Batch size 64; Input size = 300 + 512 + 256; Learning rate 0.001. Once the training of the hybrid model completed, testing for each image from test dataset is done with an input obtained by concatenating extracted 256-d features from the image using the CNN with the composite 300-d Word2Vec word-embedding and 512-d BERT word-embedding of the identified objects in the image.
4.3 RESULT
we have provided additional knowledge of objects in input images into the hybrid network incor- porating those domain nules. The hybrid network has both Word2Vec and BERT transformation

---

Under review as a conference paper at ICLR 2023
word-embedding. After the entire framework is trained and validated on the selected dataset, quan- titative results are obtained from the evaluation on the testing set. Additionally, VGGnet and MobileNet V3 is referenced as the baseline model. The experiment results show in It shows the overall classification performance of hybrid network with two word-embeddings for the testing set and of the other improvement of all metrics from VGGnetl6 and MobileNet V3 to hybrid network, which adequately demonstrates the effectiveness of our framework. Note that for each of tagged object, which is im- ported as inputs to the network, there is at least one rule with one of the conditions in the antecedent matching the object. What's more, qualitative results for some images compared baseline in the testing set are visualized in
baseline networks. Across the board, the whole figure indicates the overall
museum
church
library
Ibrary
library
library
concert
cncertt
concert
oncert
church
church
church
church
0.0
0.5
10
0.0
0.5
100
0.0
0.5
10
0.0
0.5
10
library
library
library
library
museum
museum
museum
museum
dhurch
church
dhurch
church
mall
mall
mall
mall
10
mall
library
library
lbbrary
llbrary
library
museum
concert
concert
concert
cnccrtt
church
church
church
church
0.0
0.5
100
0.0
0.5
100
0.0
0.
10
0.0
0.5
10
Figure 3: Overall improvement in prediction through base CNN to rules embedding
Table 2: Main evaluation results and baseline results
Model
Accuracy
(ecall(%)
Precision(?) (%)
F1(%)
VGGI6
43.15
42.75
40.41
37.17
MobileNet V3
37.67
40.77
36.30
34.27
Ours 73.29
74.31
73.29
73.02
4.4 PERFORMANCE EVALUATION
First, we tested the hybrid model to show that its functions like the basic 3-layer CNN when for each input image's ' symbolic part is reduced to zeros. This means for an input scene with no tagged iden- tified objects, the base CNN will function the same way the hybrid with no additional symbolic knowledge and did our case.
Second, we compared fusion of CNN image features plus automatically generated rules and fusion of CNN image features plus manual rules. The comparison results are shown in 3 As we can see, rules taken out from objects detection the former has more prior knowledge based on objects each image, while the latter has more information not be restrained in the image.
better than the human encoded ones. This reveals that
Table 3: CNN with different kinds rules
Framework
Accuracy(%))
Recall(%)
Precision(%)
F1(7)
CNN
Human Encoded
64.38
65.03
63.70
64.61
CNN + Automatically Generated
73.29
74.31
73.29
73.02
Third, we tested that only logic rules are used to predict the classification of indoor scenes. As shown in 4 fusion framework has overall better performance than nules alone. Though, human
8

---

Under review as a conference paper at ICLR 2023
encoded rules fused with CNN have performance gap rules training only. This also reveals that these automatically generated rules from images can be regarded as type of extracted feature.
Table Only logic rules
Rules
Accuracy(%}
Recall(%)
Precision(%) (%)
F1(%)
Human Encoded
34.93
34.93
34.93
26.96
Automatically Generated
73.29
73.29
73.29
68.37
Fourth, we have implemented three ablation experiments which are 3-layer CNN, 3-layer CNN plus Word2 representation and 3-layer CNN plus BERT representation, hhooss the comparison results with our main hybrid fusion approach. indicates that rules representation adding image features helps to improve classification performance. BERT representation has more influence to classification results since the BERT classifier has been trained and represents word vectors better. Futhermore, the performance of our main experiment is beyond the fusion of representations of any single type of rules vector.
Table 5: Ablation study with different word representations
Framework
Accuracy(%)
Recall(%)
Precision(%)
F1(%)
CNN
56.16
57.34
56. 16
53.04
CNN + Werd2Vec
66.44
67.36
66.44
64.02
CNN BERT
69.18
69.18
69.18
68.76
CNN Word2Vec BERT (Ours)
73.29
74.31
73.29
73.02
FUTURE WORK
We have developed hybrid modeling framework which combines CNN with propositional rules to allow integrating subjective domain knowledge, relating objects to scenes, into the neural models. Our experiment with scene classification scenario shows substantial improvement over classifica- tion by the underlying CNN model without rules. We are currently in the process of extending the hybrid modeling framework with rules expressed in Horn clauses which is subset of full first-order formulae Lloyd 1984). We are also investigating how to incorporate uncertainty associated with the output of typical object detection algorithm.
One application we are actively working on is situation and threat assessment Martin Liggins II (0088) within an area under surveillance, where intelligence comes from a variety of multimodal sources, including human and signal intelligence. Unlike scene, which is visible, a specific situa- tion or threat is a non-visible abstraction of various visible objects present in the environment. We believe subjective knowledge from a domain can help improving such assessment tasks.
Finally, Baker et al. Baker et 2018 observed that deep CNNs have access to some local shape information in the form of local edge relations. Tasks such as automated scene classification and situation assessment require learning of the presence or absence of objects and their spatial rela- tionships. Hence, the scope and usefulness of incorporating domain expert knowledge and known sensor behavior into deep CNNs is very high.
REFERENCES
Nicholas Baker, Hongjing Lu, Gennady Erlikhman, and Philip Kellman. Deep convolutional net- works do not classify based global object shape. PLOS Computational Biology, 14, December 2018. doi: fo.137 1006613.
Le- Wen Cai, Wang- Zhou Dai, Yu-Xuan Huang. Yu-Feng Li. Stephen Muggleton, and Yuan Jiang. Abductive learning with ground knowledge base. In Zhi-Hua Zhou (ed.), Proceed- ings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-2I, pp- 1815-1821. International Joint Conferences Artificial Intelligence Organization, 8 2021. doi:

---

Under review as a conference paper at ICLR 2023
00.24963//jcai22221/255 http:///w....///.///////////////
Main
Track.
Bao Xin Chen, Raghavender Sahdev, Dekun Wu, Xing Zhao, Manos Papagelis, and John Tsotsos. Scene classification in indoor environments for robots using context based word embeddings. In International Conference on Robotics and Automation (ICRA) Workshops, 05 2018.
Riquan Chen, Tianshui Chen, Xiaolu Hui, Hefeng Wu, Guanbin Li, and Liang Lin. Knowledge graph transfer network for few-shot recognition. In AAAI, 2020.
Sizhe Chen, Haipeng Wang, Feng Xu, and Ya-Qiu Jin. Target classification using the deep convolu- tional networks for sar images. IEEE Transactions on Geoscience and Remote Sensing, 54:1-12, 04 2016. doi: 1.1110/TGRS..016255511111
William W. Cohen. Tensorlog: A differentiable deductive database. 05 2016.
Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. Natural language processing (almost) from scratch. Mach. Learn. Res., 12:2493 2537, November 2011. ISSN 1532-4435.
Larry Davis, Devi Parikh, and Fei-Fei Li. Future directions of visual common sense recognition. 11 2015.
Thomas Demeester, Tim Rocktäschel, and Sebastian Riedel. Lifted rule injection for relation em- beddings. Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 1389-1399, 11 2016. doi: 10. N0.18653/v1/D166 1146.
Liya Ding, Hoon Heng Teh, Peizhuang Wang, and Ho Chung I Lui. A prolog-like inference system based neural logic- attempt towards fuzzy neural logic programming. Fuzzy Sets Syst., 82(2):235-251, September 1996. ISSN 0165-0114. doi: 10.106600655014455022244 URL 011 (95) 092509
Dttps://60i.0r//10..1000
Ivan Donadello, Luciano Serafini, and Artur D' Avila Garcez. Logic tensor networks for semantic image interpretation. In Proceedings of the 26th International Joint Conference on Artificial Intelligence, IJCAI 17, pp. 1596-16022 AAAI Press, 2017. ISBN 9780999241103.
Manoel Franca, Gerson Zaverucha, and Artur Garcez. Fast relational learning using bottom clause propositionallization with artificial neural networks. Machine Learning. 94:81-104, 01 2014. doi: 10. 1001100771109944-01--33922...
Artur Garcez, Luís Lamb, and Dov Gabbay. Neural-Symbolic Cognitive Reasoning. Springer, 2009. ISBN 978-3-540-73245-77 doi: 10.1007/978-3-540-33244
Ross Girshick. Fast r-cnn. In 2015 IEEE International Conference on Computer Vision (ICCV), pp- 1440-1448, 2015. doi: 10.1 2..1109//CC..2015.660
Alex Graves, Abdel-rahman Mohamed, and Geoffrey Hinton. Speech recognition with deep recur- rent neural networks. ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing Proceedings, 38, 03 2013. doi: 10.1 109/CCAASS...011.666.9....
Mathieu Guillame -Bert K. Broda, and Artur Garcez. First-order logic learning in artificial neural networks. pp. 1 8, 08 2010. doi: 10. 11000//CN...22000555999999999999
Helmar Gust, Kai- Uwe Kühnberger, and Peter Geibel. Learning Models Predicate Logical The- ories with Neural Networks Based on Topos Theory, volume 77, pp. 233-264. 08 2007. ISBN 978-3-540-73953-11 doi: 10. 0007/978-3-540739555
Joseph Y. Halpern. Reasoning about Uncertainty. MIT Press, 2005.
Barbara Hammer and Pascal Hitzler. Perspectives of 2007. doi: 10. 00.1007/978---500 8-4999-----
Integration, volume 77.
10

---

Under review as a conference paper at ICLR 2023
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Spatial pyramid pooling in deep convo- lutional networks for visual recognition. In David Fleet, Tomas Pajdla, Bert Schiele, and Tinne Tuytelaars (eds.), Computer Vision ECCV 2014, pp. 346-361, Cham, 2014. Springer Interna- tional Publishing. ISBN 978-3-319-10578-9
Geoffrey Hinton, li Deng, Dong Yu, George Dahl, Abdel-rahman Mohamed, Navdeep Jaitly, An- drew Senior, Vincent Vanhoucke, Phuongtrang Nguyen, Tara Sainath, and Brian Kingsbury. Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups. Signal Processing Magazine, IEEE, 29:82-97, 11 2012. doi: 10.1109/MSP.
2012.2205597.
Steffen Hölldobler. A structured connectionist unification algorithm. Proceedings of the Highth National Conference on Artificial Intelligence VVoluee AAAT'0, pp. 587-593, AAAI Press, 1990. ISBN 026251057X.
Nal Kalchbrenner, Edward Grefenstette, and Phil Blunsom. A convolutional neural network for modelling sentences. 52nd Annual Meeting of the Association for Computational Linguistics, ACL 2014- Proceedings of the Conference, 1, 04 2014, doi: 10.3115/11/14--06...
Michael Kampffmeyer Yinbo Chen, Xiaodan Liang, Hao Wang, Yujia Zhang, and Eric P. Xing. Rethinking knowledge graph propagation for zero-shot learning. In International Confer- ence on Learning Representations, 2019. URL
met/forrm?io
rkgsDoAqFQ
Ekaterina Komendantskaya, Unification neural networks: Unification by error-correction learning. Logic Journal of the IGPL, 19:821-847, 12 2011. doi: 10. (0.1093/jippaliiz001.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. Imagenet classification with deep con- volutional neural networks. Commun, ACM, 60(6):84-9,, May 2017. ISSN 0001-0782. doi:
10.1145/3065386. URL http::tt/s...ooroteeee http://i..or/...///o// 1ttp://1..111//33555533
Yann LeCun, Y. Bengio, and Geoffrey Hinton. Deep learning. Nature, 521:436-44, 05 2015. doi: 10. 10.8////r 14539.
Aoxue Li, Tiange Luo, Zhiwu Lu, Tao Xiang, and Liwei Wang. Large-scale few-shot learning: Knowledge transfer with class hierarchy. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 7205-7213, 2019. doi: 10.1109/CVPR.2199000
Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng-Yar Fu, and Alexander Berg. Ssd: Single shot multibox detector. volume 9905, pp. 21-37, 10 2016. ISBN 978-3-319-46447-3. doi: 10. 10.11007/978-3339-444444
W. Lloyd. Foundations of Logic Programming. Springer-Vetie Verlag, Berlin, Heidelberg, 1984. ISBN 038913
Robin Manhaeve, Sebastijan Dumancie, Angelika Kimmig, Thomas Demeester, and Lue De Raedt. Deepproblog: Neural probabilistic logic programming, 07 2019.
Jiayuan Mao, Chuang Gan, Pushmeet Kohli, Joshua B. Tenenbaum, and Jiajun Wu. The neuro- symbolic concept learne:: Interpreting scenes, words, and sentences from natural supervision. In International Conference Learning Representations, 2019. URL/ttps:: httpittppss//ppeeriieeess /owpereeeeeee....
net. iiniiiddiiir
d= JqM1 JgM1hhctm
Giuseppe Marra, Michelangelo Diligenti, Francesco Giannini, Marco Gori, and Marco Maggini. Relational neural machines. In 24th European Conference on Artificial Intelligence, ECAI-2020, 2020.
James Llinas Martin Liggins II, David Hall. Handbook of Multisensor Data Fusion, Theory and Practice, Second Edition. CRC Press, 2008. ISBN 978 127154000....
Tomas Mikolov, Kai Chen, G.s Corrado, and Jeffrey Dean. Efficient estimation of word representa- tions in vector space. Proceedings of Workshop at ICLR, 2013, 01 2013.

---

Under review as a conference paper at ICLR 2023
Stephen Muggleton. Inductive logic programming. New Generation Computing, pp. 295-318, 02 1990.
Jiquan Ngiam, Aditya Khosla, Mingyu Kim, Juhan Nam, Honglak Lee, and Andrew Ng. Multimodal deep learning. pp. 689-696, 01 2011.
Anh Nguyen, Jason Yosinski, and Jeff Clune. Deep neural networks are easily fooled: High confi dence predictions for unrecognizable images. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 12 2014.
Baolin Peng. Zhengdong Lu, Hang Li, and Kam-Fai Wong. Towards neural network-based reason- ing. 08 2015. htRp:t::::// http::://aaxxv...o/ awxiv.orgiioo......o..ooo.....oa abs/aaass500 05508
Luc De Raedt, Robin Manhaeve, Sebastijan Dumancic, Thomas Demeester, and Angelika Kimmig. Neuro-symbolicc neural + logical + probabílistic. In Proceedings of the 2019 International Workshop on Neural-Symbolic Learning and Reasoning, 2019.
Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi. You only look once: Unified, real-time object detection. In 2016 IEEE Conference on Computer Vision and Pattern Recognition doi: 10. 10.1109/CVPR.2066.999
(CVPR), pp. 779-788, 06 2016.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 39, 06 2015, doi: 10.1 10.109/7PAM122011557000
Matthew Richardson and Pedro Domingos. Markov logic networks. Mach. Learn., 62(1-2): 107-136, February 2006. ISSN 0885-6125.
doi: 10.1007/5109944000-53333 URL https:
/ídoi.or//00 1tt./10001/999/0 006 5833-1
Tim Rocktäschel and Sebastian Riedel. Learning knowledge base inference with neural theorem provers. In Proceedings of the 5th Workshop on Automated Knowledge Base Construction, pp. 45-50, 01 2016. doi: 10. 20.18653//1/V 1309.
Paul Rosenbloom, Abram Demski, and Volkan Ustun. Toward neural-symbolic sigma: Intro- ducing neural network learning. In Proceedings of the 15th Annual Meeting of the International Conference on Cognitive Modeling. 2017.
Juergen Schmidhuber. Deep learning neural networks: An overview. Neural Networks, 61, 04 2014. doi: 10. 00.0016f.neunee.2014.0000.0
Pierre Sermanet, David Eigen, Xiang Zhang, Michael Mathieu, Rob Fergus, and Yann Lecun. Over- feat: Integrated recognition, localization and detection using convolutional networks. Interna- tional Conference on Learning Representations (ICLR) (Banff), 12 2013.
Lokendra Shastri. Advances in shruti-a neurally motivated model of relational knowledge repre- sentation and rapid inference using temporal synchrony. Applied Intelligence, 11, 05 1999. doi: 1008330660000886111111
Jude W. Shavlik and Geoffrey G. Towell. Combining explanation-based and neural learning: An algorithm and empirical results, 1989.
Yelong Shen, Xiaodong He. Jianfeng Gao, Li Deng, and Grégoire Mesnil. A latent semantic model with wwwvollutional-pooling structure for information retrieval. In Proceedings of the 23rd ACM International Conference Conference on Information and Knowledge Management, CIKM '14, pp. 101-110, New York, NY, USA, 2014. Association for Computing Machinery. ISBN 97814 9781450325981. doi: 121145/266182.2666111 2661 829. 2661935
URL
Gustav Sourek, Vojtech Aschenbrenner, Filip Zelezny, and Ondrej Kuzelka. Lifted relational neural networks. Journal of Artificial Intelligence Research (JAIR), 08 2015.
Nitish Srivastava and Ruslan Salakhutdinov. Multimodal learning with deep boltzmann machines. In Proceedings of the 25th International Conference on Neural Information Processing Systems- Volume 2, NIPS' 12, pp. 2222-222222222222222222222222 22200, Red Hook, NY, USA, 2012. Curran Associates Inc.
12

---

Under review as a conference paper at ICLR 2023
Ron Sun. Integrating Rules and Connectionism for Robust Commonsense Reasoning. John Wiley amp; Sons, Inc., USA, 1994. ISBN 0471593249.
Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. International Conference on Learning Representations, 12 2013.
Stefan Wermter and Ron Sun. The present and the future of hybrid neural symbolic systems: Some reflections from the nips workshop. Al Magazine, 22:123-128 03 2001.
Yaqi Xie, Ziwei Xu, Mohan Kankanhalli, Kuldeep S. Meel, and Harold Soh. Embedding Symbolic Knowledge into Deep Networks. Curran Associates Inc., Red Hook, NY, USA, 2019.
Jingyi Xu, Zilu Zhang, Tal Friedman, Yitao Liang, and Guy Van den Broeck. A semantic loss function for deep learning with symbolic knowledge. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 5502-5511. PMLR, 10-15 Jul 2018. URL htttps:/pproceedingg ml prese xu1 1 Bh html.
Yuan Yang and Song. Learn to explain efficiently via neural logic inductive learning. In Interna- fional Conference Learning Representations, 2020. URL https:: /ttps:://penreeieew net.
Sorum?id-SS1h8CEVV
Wen-tau Yih, Ming-Wei Chang, Xiaodong He, and Jianfeng Gao. Semantic parsing via staged query graph generation: Question answering with knowledge base. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Confer- ence on Natural Language Processing, pp. 1321-1331, 01 2015. doi: 10.3115//1/155-1122
Eloi Zablocki, Patrick Bordes, Laure Soulier, Benjamin Piwowarski, and Patrick Gallinari. Context- aware zero-shot learning for object recognition. In Kamalika Chaudhuri and Ruslan Salakhutdi- nov (eds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 7292-7303, PMLR, 09-15 Jun 2019. URL https::/prroceeding. mww...s.........mm.............. press wttpee///////aalc/....ccc htm.lmmmmmmmmmm
Yuyu Zhang, Xinshi Chen, Yuan Yang, Arun Ramamurthy, Bo Li, Yuan Qi, and Le Song. Efficient probabilistic logic reasoning with graph neural networks. In International Conference on Learn- ing Representations, 2020.
net/ htte:/ooommnddduggeee
Xiaoxiang Zhu, Devis Tuia, Lichao Mou, Gui-Song Xia, Liangpei Zhang, Feng Xu, and Friedrich Fraundorfer. Deep learning remote sensing: comprehensive review and list of resources. IEEE Geoscience and Remote Sensing Magazine, 5:8-36, 2017.
13


Reviews:
[{"review_id": "d_0s-j-f78V", "reviewer": "Reviewer_vv79", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes to combine symbolic representation and distributed representation together to improve image classification. The proposed model first uses a pre-trained fast RCNN to perform object detection on every example, and then automatically construct a rule set, in which every rule is a combination of the discovered objects in one example. Then, these rules are used for extract relational features from training examples, which will be fed into a seq2seq deep learning model. Combined with the CNN features, the final feature vector will be further processed by an MLP to make the final prediction.", "summary_of_the_review": "This paper in its current condition is not ready for publication.", "strength_and_weaknesses": "### Strengths:\n- This paper targets at solving an important problem, which is highly related to this conference.\n\n### Weakness:\n- The presentation of this paper can be significantly improved, there are many grammatical errors that need to be revised. The only one equation is not numbered and is unexplained, and many descriptions are confusing. Apparently, this work needs proofreading before submission and is not ready for publication now.\n- The authors claim that the proposed approach aims at bridging neural and symbolic models, however, the symbolic representation is converted into embeddings again. Since relational features are extracted by a pretrained fast RCNN, why not directly attach the RCNN to the final predictive MLP? What is the purpose of extracting combinations of objects as rules?\n- The hyper-parameters on rule confidence seem arbitrary.\n- The experiments are weak, no state-of-the-art scene classification models were compared.\n- There are some works trying to learn visual concepts using combinations of sub-concepts in a symbolic form, for example:\n  - Zhongyi Han, Le-Wen Cai, Wang-Zhou Dai, Yu-Xuan Huang, Benzheng Wei, Wei Wang, Yilong Yin, “Abductive Subconcept Learning.” Science China Information Sciences, 2022.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "POkNGrLw-tO", "reviewer": "Reviewer_jnEW", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a complex design to classify   Images: it   involves w2v, bert, user defined and bayesian features, that are sent to DNN,,\nResults look good., but on the author'ś data.\n", "summary_of_the_review": "The paper needs to give us a more formal presentation and evaluation.\n\nMore in detail;the key idea of the paper consists in feeding high--level attributes to a DNN. This leads the authors to a 3 layer arch:\n\n1. The  first step is to run  a cnn and Fast-Run CNN to obtain high-level features from the images. This is followed by a filtering step. It is not clear if the parameters were chosen ad-hoc or if  there was some kind of search, It is also not clear whether the new features are fed to the DNN, or just used by step 2.\n\nThis is also not clear in Fig 2, which ssuggest the CNN runs in parallel with Fast CNN. I got a different impression from the text.\n\n2, These features may be used by:\n- probabilistic rules obtained by what seems like a variation of a Priori.It woould be useful to have more  details on how  you compute the parameters. Also, you may want to look into softening to avoid 100% probabilities.\n\n- User-written rules: not much detail is given, Table2 might help, f it was there ?\n\n3. you use w2vec, bert and a singke-layer perceptron> Why these choices?\n\nResults¨\n\n- related systems: why these? How were they trained?\n\nThe english needs work, 3.4.2 is a particularly bad ex.\n\nFinally, i was hoping, but I failed;\n to understand why your model works\n to see other applications\n\n", "strength_and_weaknesses": "Strengths\n it merges important ideas\nthe results are good.\nWeaknesses¨:\nWhy? Often decisions are not clearly explained, eg Bert +/vs w2vec\nThe probabilistic rules deserve an example.\nThere is little info on how the system is configured\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "PFXlpiz0KVJ", "reviewer": "Reviewer_XLX8", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors introduce a neuro-symbolic image classifier based on feature fusion.  The idea is to glue together (1) the image representation provided by embedding layers with (2) BERT and Word2Vec embeddings of (a textual representation of the body of Horn) rules that match objects detected in the image, and then (3) feed the resulting vector to a neural classifier.  The two embedding approaches are chosen so as to capture different types of information, namely structural and semantic/conceptual, intuitively complementing each other.  The proposed approach is compared against VGG16 and MobileNet V3 on a single indoor scene classification data set.", "summary_of_the_review": "The intuition behind the method might have some potential, but the text is not up-to-par and the experiments are severely lacking.", "strength_and_weaknesses": "PROS\n- Proposed method outperforms basic image models on the selected data set.\n- Ablation experiments.\n\nCONS\n- Text is readable, but contains a number of grammatical mistakes.\n- Method description not clearly distinguished from experimental details.\n- Model is not formalized properly.\n- Very limited empirical comparison on a single task against non-SOTA, non-NeSy baselines.\n- Related work is quite comprehensive but poorly structured.\n- Overall, the work generally feels incomplete and rushed.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "I8S3Hbw9Cbb", "reviewer": "Reviewer_fQgQ", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "A neuro-symbolic approach is proposed where a convolution neural network is extended with structured if-then symbolic rules based on word embeddings to improve image classification.", "summary_of_the_review": "The paper explores a nice and simple idea, but the approach is potentially too complicated and heavy-engineered and the actual technical novelty is quite limited. This is more of an application paper than one that studies core new research. The experiments are also limited both in terms of dataset explored and baselines compared to. I therefore at this point unfortunately recommend rejection but encourage the authors to keep working on the paper to improve the discussed aspects.", "strength_and_weaknesses": "**Strengths**: \n* **Motivation and general idea**: The paper does well in motivating the potential benefits of combining learned classification with crispier logical rules and the introduction is completing and setting the right content and necessary background. \n* **Results**: The proposed approach achieves improved results compared to the baseline.\n\n**Weaknesses**:\n* **Convoluted and heavy-engineered embedding approach**: The pipeline that combines object detection, BERT and word2vec makes the approach unnecessarily complicated and weakens the main idea of the paper, since it becomes unclear whether the empirical improvements in performance come from the approach itself or since we use multiple pretrained models that had access to further strong object-level and textual supervision. Would be good both to have more experiments to justify the contribution of each component, and look for ways to simplify the approach. The most obvious one is the use of both BERT and Word2Vec -- is that really necessary or maybe the reliance on two parallel approaches could be eliminated? Are there ways to maybe e.g. combine representations from different layers of BERT to avoid using also Word2Vec?\n* **Usage of heuristics**: For instance, in the object-detection section, the paper uses a heuristic to avoid duplicated objects by counting them. But faster R-CNN gives many potentially overlapping predictions with different degrees of confidence. Extracting an object count using such heuristic may not work effectively. Likewise, the arbitrary probabilities given to objects, selected by a person, are unjustified, and the need to use rules made by hand might greatly limit the robustness and applicability of the approach to different domains and distributions\n* **No use of standard datasets**: The paper uses a custom dataset rather than standard ones for a task where there could be plenty of public datasets that could be explored. This is a major weakness and this choice is unjustified in the paper either. The constructed dataset is also very small (733 images (440/146/146 for training/validation/testing)), weakening the statistical strength of the empirical results.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject