DEIM Forum 2018 E7-3

Classifying Community QA Questions
That Contain an Image
Kenta TAMAKI† , Riku TOGASHI†† , Sumio FUJITA†† , Sosuke KATO† , Hideyuki MAEDA†† , andTetsuya SAKAI†† Waseda University,
3–4–1, Shinokubo, Tokyo, 169–0072, Japan†† Yahoo Japan Corporation,
1–3, Kioityo, Tokyo, 102–8282, Japan
E-mail: †madao@akane.waseda.jp, sow@suou.waseda.jp, tetsuyasakai@acm.org,††{rtogashi,sufujita,hidmaeda}@yahoo-corp.jp
Abstract We consider the problem of automatically assigning a category to a given question posted to a Community Question Answering (CQA) site, where the question contains not only text but also an image. For example,
CQA users may post a photograph of a dress and ask the community “Is this appropriate for a wedding?” where
the appropriate category for this question might be “Manners, Ceremonial occasions.” We tackle this problem using
Convolutional Neural Networks with a DualNet architecture for combining the image and text representations. Our
experiments with real data from Yahoo Chiebukuro and crowdsourced gold-standard categories show that the DualNet approach outperforms a text-only baseline (p = .0000), a sum-and-product baseline (p = .0000), Multimodal
Compact Bilinear pooling (p = .0000), and a combination of sum-and-product and MCB (p = .0000). where thep-values are based on a Tukey Honestly Significant Diﬀerence test with B = 5000 trials.
Key words Classification, Convolutional Neural Network, Community Question Answering

1. Introduction
In Community Question Answering (CQA) sites, posted

cally important: in the real CQA data used in our study,
approximately 10.2% of the questions actually contain animage.

questions are organised by category so that users can find

We tackle the aforementioned classification problem using

their desired questions easily. Each CQA site typically has

Convolutional Neural Networks (CNNs) [5] with a DualNet

its own hierarchy of mutually exclusive categories, where thearchitecture for combining the image and text representa-

top-level categories might be “Entertainment and Hobbies,”

tions. Using real data from Yahoo Chiebukuro, we conduct“News, Politics, International aﬀairs,” and so on. In Yahooa large-scale question classification evaluation where the cat-

（注 1）

Chiebukuro, the most widely-used CQA site in Japan,

egories actually assigned by the questioners are considered towhen a questioner posts her question, the site presents her

be the gold standard, and a smaller-scale experiment wherewith an automatically selected list of possible categories,

the gold standard is constructed based on the views of crowd

from which she can select one and tag her own question

workers.

with it. Our goal is to automate the problem of assigningapproach outperforms a text-only baseline (p = .0000), a

a top-level category to a given question, where the ques-

sum-and-product baseline (p = .0000), Multimodal Compact

tion contains not only text but also an image. For example,

Bilinear pooling (p = .0000), and a combination of sum-

CQA users may post a photograph of a dress and ask theand-product and MCB (p = .0000). where the p-values arecommunity “Is this appropriate for a wedding?” where thebased on a Tukey Honestly Significant Diﬀerence test withappropriate category for this question might be “Manners,

B = 5000 trials.

Ceremonial Occasions.” Clearly, this task is more challenging than classifying purely textual questions, and is practi-

The latter experiment shows that the DualNet

2. Related Work
2. 1 Back Ground（注 1）：https://chiebukuro.yahoo.co.jp/

Handling information from multiple modalities has been

studied in various fields. Among these studies, Visual Question Answering (VQA) [1], [2], visual grounding [2], [7], andimage captioning [6] are closely related to our task in thatthese tasks all involve handling of both image and text. In
VQA, the system is given an image and a question aboutthat image, and is required to output an answer in natural
language. In visual grounding, the system is given an image
and a natural language description, and is required to return
a bounding box within that image that corresponds to the

Figure 1 Image of DualNet for real images category

description. In image captioning, the system is given an image, and is required to output a natural language descriptionof that image.
Of the above, VQA and visual grounding, require the understanding of both image and text inputs. However, notethat the input text in these tasks concerns whatever is featured within the input image: for example, an input text for

Figure 2 Proposed CNN architecture.

visual grounding may be “(locate) a small white dog (within
the picture)” [7]. In contrast, our question classification task
involves question texts that generally provide a context outside the input image: for example, in the aforementioned example with a photograph of a dress, the accompanying question “Is this appropriate for a wedding?” does not describe
any feature within that photograph; rather, it complements
the information conveyed in the image, while referring to the

Figure 3 Character level convolution for the text classification
network.

dress with the demonstrative pronoun “this.” In short, ourclassification task is diﬀerent from the aforementioned tasks

we propose to use a model pre-trained with images actu-

that involve both text and images.

ally posted to Yahoo! Chiebukuro, along with a pre-trained

2. 2 Multimodal Compact Bilinear poolingIn the context of VQA and visual grounding, Fukui et

ResNet model.

3. Proposed Classification Methods

al. [2] proposed Multimodal Compact Bilinear (MCB) pooling for joint representation of image and text. The Com-

Figure 2 shows our proposed CNN architecture for ques-

pact Bilinear pooling model [3] is a technique to compress

tion classification.

the high-dimension output of a traditional bilinear poolingleft) is a simple 5-layer CNN, where the input image size

The image classification network (topmodel [10]; MCB is a multimodal version of this technique.

is 128*128, with a mini batch size of 64. Each convolution

Fukui et al. [2] argue that MCB complements basic opera-

layer applies convolution, batch normalization, ReLU acti-

tions such as vector concatenation. In the present study, we

vation, and max pooling. Dropout is applied in the fully

apply MCB to the problem of question classification whereconnected layer.

each question contains text and an image. Moreover, we

The text classification network (bottom left) is a character-

propose a combination of MCB with the simple sum andlevel CNN [11]. Figure 3 depicts our character-level convolu-

element-wise product approach.

tion method. The input text size was set to 121, which was

2. 3 DualNet

the average number of characters in our training data. Ques-

DualNet [8] was proposed in 2016 as a method of VQA,

tions whose text part was longer than 121 characters wereand it outperfomed the state of the art in VQA Challenge

excluded from our experiment. The vocabulary size (i.e.,

2016. Figure 1 depicts the DualNet method in the real im-

the number of distinct characters) of the training data wasages category. In this method, image and text information

5,206; accordingly, we set the size of the embedding matrix

are combined by fusing the last hidden layer of some pre-to 5,208*embedding-size (namely, 200), after adding “whitetrained models. For the real images task, they obtained thespace” and “unknown” to the vocabulary. Questions in the

text representation from the LSTM’s last hidden layer, andvalidation and test data that were shorter than 121 charac-

the image representation from the hidden layer of multiple

ters were stuﬀed with white spaces; Characters that never ap-

pre-trained models (such as ResNet [4]). In the present study,

peared in the training data were treated as “unknown.” Ourtext classification network first prepares an embedding ma-

Table 1 Number of questions in our CQA data.

trix that consists of distributed representations corresponding to each character that can be input. Next, we computea convolution with a 200*height-size*#channel kernel to thematrix obtained by concatenating the distributed representations of the input characters.
Our final network (Figure 2 middle) utilises pre-trainedmodels of both of the aforementioned networks and pretrained ResNet [4] network. Using the hidden layers fromthe image and text networks, we consider the following four
methods for combining the image and text representations

all questions

11,074,960

questions with an image
used in the experiment693,519

training questions

554,777

test questions

69,355test questions
with crowdsoured category labels
validation questions

Japanese).

SP Calculate the Sum and element-wise Product; concate-

Manners, Ceremonial occasions

nate the results and pass it to the fully connected layer;Entertainment and Hobbies

tation (256*256 → 2,048); then pass the joint representation

5,190
69,387

Table 2 Ten categories used in our experiment (originally in(Figure 2 right):

MCB Use MCB to generate a joint image and text represen-

1,128,167

questions with an image

News, Politics, International aﬀairs
Internet, PCs, Home appliances
Life and romance, Worries of human relations

to the fully connected layer;

Life, and Living guide

SP+MCB Combination of the above two. As shown in Fig-

Health, Beauty, Fashionure 2, the MCB representation goes through a fully con-

Liberal Arts, Learning, Science

nected layer and then is concatenated with the Sum andSports, Outdoor,Cars

element-wise Product. The result of concatenation is thenRegion, Travel, Outing

passed to another fully connected layer.

we can compute the classification accuracy for each of our

DualNet Calculate the sum and element-wise Product of

methods. Note that, since our problem setting considers ten

text representation, image representation and the hidden

top-level categories, a random system would only achieve alayer of a pre-trained ResNet model with tanh activation,

classification accuracy of 10% on average.

and concatenate the results. The processing of this method is

We argue, however, that the above approach of utilising

shown below (I1 = image representation from our image clas-

the actual question category as the gold-standard is not nec-

sification network, I2 = image representation from ResNet, Q

essarily the best way to evaluate our classification systems.

= text representation from our text classification network).

This is because diﬀerent people may have diﬀerent views
about which category is most appropriate to a given question, and the category assigned by the questioner may not

I1′ = tanh(WI1 I1 )(1)

be the same as the one a CQA user, who is looking for an

I2′ = tanh(WI2 I2 )(2)existing question by category, might choose. As the goal

Q = tanh(WQ Q)(3)of our automatic question classifiers is to provide quick ac-

FS = I1′ + I2′ + Q′

(4)

FP = I1′ ◦ I2′ ◦ Q′

(5)

F = Concat(FS , FP )(6)′cess to CQA users, the views of the CQA users may in fact
be more important than that of the questioner. Moreover,the category assigned by the questioner reflects one person’s
point of view: the above evaluation methodology cannot takemultiple viewpoints into account.

4. CQA and Ground Truth Data

Based on the above argument, we constructed a set of

To evaluate our question classification methods, we used

crowdsourced category labels for a subset of the above test

real data from Yahoo Chiebukuro. Table 1 shows some statis-

questions. Due to a budget and time constraint, we randomly

tics of our data: of the approximately 11M questions, ap-

selected 519 questions from each category, and thereby con-proximately 1.12M (10.2%) contains an image; from this set,

structed a set of 5,190 questions for this second experiment.

we extracted 693,519 image-attached questions that coverFor each of these questions, five crowd workers were assigned,

the ten major top-level categories shown in Table 2. For our

who independently labeled the questions with a category.

experiments, this set was divided into training, test, and val-

The crowd workers were shown a test question contain-

idation data sets with the 8:1:1 ratio, and as a result, we ob-

ing an image, and were asked to select the most appropri-

tained 69,355 test questions. By treating the category that is

ate category from the list shown in Table 2. They were in-already attached to each test question as the gold standard,

structed that the purpose of assigning a category was to en-able quick access to the desired question on a CQA site. A

5. 2 Evaluation by Questioners’ Categories

total of 205 crowd workers participated in constructing the5, 190 ∗ 5 = 25, 950 category labels.
Rather than deciding on one true category based on a majority vote, we fully utilised the labels from the five assessors
as follows. Let C be the set of categories, and let ci ∈ C be
the i-th category. Let votes(q, ci ) be the number of crowd
workers (<= 5) who assigned ci to a given test question q. If
the system assigns a category ci to q, we give votes(q, ci )/5
points to the system for q. For example, if the system agrees
with two assessors for q, the partial score for q is 2/5 = 40%.

5. Experimental Results
5. 1 Overall Results
Figure 4 Heapmap of DualNet architecture results, where theTable 3 Experimental results. (a) is based on the correct categories as defined by the questioners. (b) is based on thehorizontal axis represents the true category as definedby the questioners.

gold labels obtained from the crowd workers.
(a) Accuracy (%) (b) Mean partial score(69,355 questions)ns.)(5,190 questions)image only

54.19

-

text only

73.48

0.5114SP

77.37

0.5165MCB

78.00

0.5193SP+MCB

78.16

0.5210

DualNet82.86

0.5411

Table 4 p-values / eﬀect sizes (standardised mean diﬀerenceswith VE = .0419 [9]) for the diﬀerences in mean partial
scores.
text onlySP

MCB

SP+MCB

DualNetp = .7140

p = .2810

p = .1150

p = .0000

Figure 5 Heapmap of text-only architecture results, where theES = .0248 ES = .0386 ES = .0471 ES = .1449

horizontal axis represents the true category as definedSP-

p = .9652

p = .7924

p = .0000

by the questioners.

ES = .0137 ES = .0222 ES = .1201
MCB

-

-

p = .9942

p = .0000

ES = .0085 ES = .1063
SP+MCB---p = .0000
ES = .0979

Figures 4 and 5 show confusion matrices with the ten categories for the DualNet architecture results and the text-onlyresults, respectively, in terms of heatmaps. The horizontal
and vertical axes represent the true (as defined by the ques-

Table 3 summarises our experimental results. It can betioners) and the predicted categories, respectively, and theobserved that, in terms of both classification accuracy andnumber of questions in each cell has been normalized by themean partial score, the performance improves as we move

number of correct questions for each cateogory. Thus, cells

down the table.

with light colors are those containing many questions. It canTable 4 shows the p-value for the diﬀerence in mean par-be observed that the true and predicted categories are more

tial score for each system pair (excluding the substantiallywell-aligned with DualNet. In particular, we can see that theunderperforming image-only baseline) based on Tukey HSD

number of questions misclassified into Category 1 (“Enter-(Honestly Significant Diﬀerences) test with B = 5000 trials,

tainment and Hobbies”) has been reduced compared to thealong with eﬀect sizes (standardised mean diﬀerences) [9].

text-only case.

It can be observed that the DualNet approach outperforms

5. 3 Evaluation by Crowdsourced Categories

a text-only baseline (p = .0000, ES = .1449), a sum-and-In this section, we evaluate the systems with crowdsourced

product baseline (p = .0000, ES = .1201), MCB poolinggold standard data. To examine the discrepancy between(p = .0000, ES = .1063), and a combination of sum-and-the questioners’ categories and the crowd workers’ majority

product and MCB (p = .0000, ES = 0.0979).

vote categories, we first computed the mean partial score

by regarding the latter as the system’s output; this gave us
0.5956, which is higher than the systems’ scores shown in Table 3, but not very high. Moreover, Cohen’s kappa between
the questioners’ categories and the crowd workers’ majority
vote categories is 0.5984, with a 95% confidence interval of[0.5839, 0.6129]. Thus, there is indeed a discrepancy between
the questioner and the crowd workers, and hence it is also
possible that there will be a discrepancy between the questioners’ categories and the expectations of the CQA users.
Table 5 An example question with the category assigned by thequestioner and that by the crowd workers
Question

What is the inner diameter(translatedof the circular barbell pierce

from Japanese)that Stav Strashko on his septum?

Questioner’s

Category 6

Category(“Health Beauty, Fashion”)

Majority vote

Category 1

from crowd workers (“Entertainment and Hobbies”)

Table 5 provides an example from the classification results.
This question asks about the inner diameter of a pierce. But
the crowd worker selected Category 1 (“Entertainment andHobbies”), probably because the question mentions “StavStrashko,” a model who was popular at that time.

6. Conclusions
Our experiments with real data from Yahoo Chiebukuro
and crowdsourced gold-standard categories show that the
DualNet approach outperforms a text-only baseline (p =.0000), a sum-and-product baseline (p = .0000), Multimodal
Compact Bilinear pooling (p = .0000), and a combination ofsum-and-product and MCB (p = .0000). where the p-values
are based on a Tukey Honestly Significant Diﬀerence testwith B = 5000 trials. Thus, while these eﬀects are small,the DualNet approach appears to be the most promising for
combining image and text representations for question classification.
Figures 4 and 5 shows both methods tend to misclassify
into Category 1. Category 1 contained the highest numberof questions while Category 0 (“Manners, Ceremonial occasions”) contained the fewest. In future work, we would liketo take countermeasures against such imbalance. Moreover,as we have observed that the categories given by questioners
and those given by crowd workers often disagree, we would
like to try training our networks from crowdsoured labels
instead of the questioners’.

References[1] Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret
Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi
Parikh. VQA: Visual question answering. In Proceedingsof the IEEE ICCV 2015, pp. 2425–2433, 2015.
[2] Akira Fukui, Dong Huk Park, Daylen Yang, Anna
Rohrbach, Trevor Darrell, and Marcus Rohrbach. Multimodal compact bilinear pooling for visual question answering and visual grounding. CoRR, Vol. abs/1606.01847, ,2016.
[3] Yang Gao, Oscar Beijbom, Ning Zhang, and Trevor Darrell.
Compact bilinear pooling. In Proceedings of IEEE CVPR
2016, 2016.
[4] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.
Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and patternrecognition, pp. 770–778, 2016.
[5] Alex Krizhevsky, Ilya Sutskever, and Geoﬀrey E Hinton.
Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems, pp. 1097–1105, 2012.
[6] Margaret Mitchell, Xufeng Han, Jesse Dodge, Alyssa Mensch, Amit Goyal, Alex Berg, Kota Yamaguchi, Tamara
Berg, Karl Stratos, and Hal Daumé III. Midge: Generating image descriptions from computer vision detections. In
Proceedings of the 13th Conference of the European Chapter of the Association for Computational Linguistics, pp.
747–756. Association for Computational Linguistics, 2012.
[7] Anna Rohrbach, Marcus Rohrbach, Ronghang Hu, Trevor
Darrell, and Bernt Schiele. Grounding of textual phrases
in images by reconstruction. CoRR, Vol. abs/1511.03745, ,2015.
[8] Kuniaki Saito, Andrew Shin, Yoshitaka Ushiku, and Tatsuya Harada. Dualnet: Domain-invariant network for visualquestion answering. In Multimedia and Expo (ICME), 2017
IEEE International Conference on, pp. 829–834. IEEE,2017.
[9] Tetsuya Sakai. Statistical reform in information retrieval?
SIGIR Forum, Vol. 48, No. 1, pp. 3–12, 2014.
[10] Joshua B. Tenenbaum and William T. Freeman. Separating style and content with bilinear models. Neural Comput.,Vol. 12, No. 6, pp. 1247–1283, 2000.
[11] Xiang Zhang, Junbo Zhao, and Yann LeCun. Characterlevel convolutional networks for text classification. In Advances in neural information processing systems, pp. 649–
657, 2015.