Understanding the Role of Images on Stack Overflow
Dong Wang† , Tao Xiao‡ , Christoph Treude§ , Raula Gaikovina Kula‡ , Hideaki Hata¶ , Yasutaka Kamei†
Kyushu University, Japan†
Nara Institute of Science and Technology, Japan‡
University of Melbourne, Australia§
Shinshu University, Japan¶
Email: d.wang@ait.kyushu-u.ac.jp, tao.xiao.ts2@is.naist.jp, christoph.treude@unimelb.edu.au,raula-k@is.naist.jp, hata@shinshu-u.ac.jp, kamei@ait.kyushu-u.ac.jp

quality questions [2–4]. The literature shows that a significant
percentage of the questions on Stack Overflow are extremely
poor in nature [5]. Consequently, these poor-quality questions
are less likely to receive answers and get solved, affecting
users’ experience. For example, recent work by Yazdaninia etal. [6] found that 47% of the Stack Overflow questions had
not received accepted answers.
Prior work has reported that providing information in a waythat promotes readability and comprehension is regarded as
the strongest indicator of the quality of the question [7]. Apartfrom the regular text-based elements (question tags, titles, anddescriptions), many studies have been carried out to understandcode snippets and investigate the role of the presence of codesnippets in question quality [8, 9]. For example, Asaduzzaman
et al. [10] observed that question categories that contain a codesnippet have a high answer ratio and may have more than one
possible good answer. Duijn et al. [7] demonstrated that quality
questions need quality code snippets. However, image, anothervisual communication method to convey information, is notwidely studied in the Stack Overflow domain.
Previous research on images on Stack Overflow [1] hasbeen conducted, but it does not consider the role of images in
I. I NTRODUCTION
asking a question and how it is answered. To address this, we
Software developers handle a wide variety of artifacts, conducted an empirical study based on 1,426,980 questions
ranging from source code and documentation to diagrams that contain 2,026,090 images. More specifically, we set out
and user interfaces. While the former two are well-suited to to characterise the images, investigate how these images are
be represented as text, the latter ones often require visual used in the questions, and quantitatively analyze their role inrepresentation in the form of images. Unsurprisingly, the receiving answers. We formulate the following three researchnumber of images used in electronic developer communication questions to guide our study:• (RQ1) What are the characteristics of images used insuch as the question-and-answer forum Stack Overflow is
Stack Overflow questions?
constantly rising. Stack Overflow allows users to attach
Motivation: The survey by Nayebi [1] stated that develadditional files along with their posted content.1 Nayebi [1]opers are increasingly sharing images in social coding
reported a linearly increasing trend in the number of posts
environments (e.g., Stack Overflow). However, it remains
including an image on Stack Overflow in the past five years.
empirically unclear what characteristics these images
Developers often rely on Stack Overflow to search forhave (e.g., source, content, and purpose). Analyzing theirvarious information on the Web. Stack Overflow as a large
characteristics will lay the foundation for understandingknowledge base has accumulated millions of programmingthe role of images as complementary pieces of information
related questions and answers. Effectively obtaining answers
when asking questions.
is essential for developers, which could further save their time
Results: In terms of the image source, we find thatand effort on software development. However, several studies
the desktop application is the most common image
pointed out that it is not trivial for developers to formulate goodsource (47%), followed by the web browser (26%). In
1 https://stackoverflow.blog/2010/08/18/new-image-upload-support/
terms of image content, we observe that the images

Abstract—Images are increasingly being shared by softwaredevelopers in diverse channels including question-and-answer
forums like Stack Overflow. Although prior work has pointed outthat these images are meaningful and provide complementary
information compared to their associated text, how images areused to support questions is empirically unknown. To address
this knowledge gap, in this paper we specifically conduct anempirical study to investigate (I) the characteristics of images,(II) the extent to which images are used in different question
types, and (III) the role of images on receiving answers. Our
results first show that user interface is the most common image
content and undesired output is the most frequent purpose forsharing images. Moreover, these images essentially facilitate theunderstanding of 68% of sampled questions. Second, we findthat discrepancy questions are more relatively frequent comparedto those without images, but there are no significant differences
observed in description length in all types of questions. Third,
the quantitative results statistically validate that questions withimages are more likely to receive accepted answers, but do not
speed up the time to receive answers. Our work demonstrates
the crucial role that images play by approaching the topic from a
new angle and lays the foundation for future opportunities to use
images to assist in tasks like generating questions and identifying
question-relatedness.
Index Terms—Q&A-forums, Stack Overflow, Images

frequently display user interfaces (58%), while they are
Stack overflow questions
rarely used to depict source code and configurations
Image ratio
With Image(5% and 2%, respectively). Three purposes for attaching
Without Image
images are identified, with the undesired output being themost commonly used (45%). From our manual analysis,
we observe that 68% of images are essential to theirsurrounding Stack Overflow questions, i.e., these questions
cannot be understood without the images.
• (RQ2) How are images used in questions?
Motivation: Existing work has studied the distribution oftypes of general Stack Overflow questions, with nine types
being classified (e.g., how-to, discrepancy, and error) [11].
(a) The proportion of SO questions that have images in an interval of aConsidering the nature of visuals, we assume that images
year.
could be utilized in the specific question types by users.
Answering this research question would help users gaina better understanding of the extent to which images areused in different question types.
Results: The results demonstrate that how-to, discrepancy,and error are the three most frequent types of questions
that contain images. Moreover, discrepancy questions
are relatively more frequent compared to the questions
without images, with an increase of 6%. In addition, ourstatistical test shows that there is no significant difference
in description length between questions with images andthose without images, but images tend to be used by(b) The popular tags of SO questions that have images.
experienced users.
• (RQ3) How are questions with images answered?
Fig. 1. Descriptive analysis of the popularity of SO questions that have images.
Motivation: To efficiently obtain accepted answers is beneficial for users searching for crowdsourcing knowledge
on Stack Overflow [6, 8]. Several works have analyzed 2 describes the data preparation. Sections 3–5 present thethe factor of various non-textual elements contained in experiments that we conducted to address RQ1–RQ3 withquestions (e.g., URLs and code snippets). For example, their results, respectively. Section 6 discusses the implications
Calefato et al. [12] found that the presence of code and challenges of our study. Section 7 situates this paper
snippets is positively associated with successful questions. with respect to related work on Stack Overflow and nonHowever, the effect of the image, a popularly used textual information sharing in software development. Section
non-textual element in modern society, remains largely 8 discloses threats to the validity of our study. Finally, we
unknown. Thus, we would like to explore whether or not conclude the paper in Section 9. To facilitate replication andthe presence of images also plays a role in the answer future work in the area, we have prepared a replication package
outcome.
online, which includes all manually labeled datasets [13].
Results: Our statistical results show that questions with
II. DATA P REPARATION
images are significantly more likely to receive accepted
In this section, we present the preparation of the data
answers significantly, including the common question
types when compared to the questions without images. consisting of the collection of image data and the descriptionof the dataset.
Images do not speed up the time to receive answers.
Data collection. To understand the role of images in StackIn summary, our contributions are three-fold: first, we Overflow questions, we mined Stack Overflow using theconstruct a taxonomy of image sources, contents, and purposes SOTorrent dataset as of December 2019 [14]. Table I showsin the Stack Overflow domain using manual classification, a summary of our studied dataset. The dataset consists ofwhich could be applied to future large-scale studies employing 18,699,426 questions. To extract questions that contain images,
automatic classifiers; second, our study sheds light on the we use regular expressions to match hyperlinks in question
usage of images in different question types by mapping image bodies. Specifically, we use a list of image file extension
characteristics and question characteristics; third, the empirical types (i.e., JPG, PNG, GIF, WEBP, TIFF, PSD, RAW, BMP,
results highlight the role of images in understanding the HEIE, INDD, JPEG, SVG, AI, EPS, and PDF) to retrieve
questions and receiving answers, complementing knowledge image-related links. Using this method, we were able to obtain
regarding the question quality.
1,426,890 questions that contain 2,026,090 images, which
The remainder of this paper is organized as follows. Section represents approximately 7% of all questions.

2interval, and confidence level is well established [17]. Based on
the calculation2 , we randomly sample 384 developer questions
that contain at least one image.
Manual coding. The qualitative analysis to understand image
characteristics was conducted in multiple rounds following asystematic method similar to the work of Hata et al. [15]. In
the first round, the first three authors independently coded 30
images from the representative sample, discussed a common
coding guide, and tested this coding guide on another 10
images, refining the guide, merging codes, and adding codesthat had been missed. In the second round, to evaluate thelevel of agreement of our constructed codes, we calculated the
Kappa agreement of our codes among the first three authors,
who independently coded (a) the source, (b) the content, (c)the purpose, and (d) the essentialness of the images of another30 randomly selected samples. The free-marginal Kappa scoresfor the content are 0.88 or ‘Almost perfect’ agreement, while
the Kappa scores for the image source and image purpose
are 0.75 and 0.71, respectively, which indicates ‘Substantial’
agreement [18]. However, the Kappa score for the essentialnessof the images was not yet sufficient. Thus, in the third round,
we refined our definition through discussion and randomly
selected another 20 samples to test the refined coding criteria
for the essentialness of the images among the first three authors.
As a result, the Kappa score for the essentialness of the image
reached 0.9 or ‘Almost perfect’ agreement. Based on thisencouraging result, the remaining data was divided into thethree sets equally and then coded by the first three authors
independently. Note that one question may contain more than
one image; however, in our study, we only manually classify
the first image that appears in the post as we assume thatimages used in a question are likely to present one specifictopic. In other words, a total of 384 images were classified
during our qualitative analysis.

TABLE IS UMMARY OF THE STUDIED DATASET
Studied Period# Questions# Questions Contain Images# Total Images# Images per Posts (1st Qu./Median/3rd Qu.)2008.06 – 2019.12
18,699,426
1,426,890 (7%)
2,026,090
1/1/2

Dataset Description. Before we address each researchquestion, we briefly characterise the popularity of questions that
contain images. To do so, we conduct two analyses to visualize(i) the trend of questions that contain images and (ii) populartags of the questions that contain images. For trend analysis,
we measure the proportion of questions that contain at least one
image in the question body in an interval of one year. For the
tag analysis, we first extract the top 10 tags sorted by the total
number of questions that contain images. Then we measure
the proportion of questions that contain at least one image for).
each of these top-10 tags (i.e., Questions_with_image_tag
T otal_Question_tag
Figure 1 (a) shows the trend of Stack Overflow questions
that contain images during our studied time span. The figure
shows that the proportion of questions that have images outof all questions generally follows an upward trend. Especially,
from 2016, the proportion of image-based questions is steadily
greater than 10%. Figure 1 (b) shows the top 10 tags that arelabeled with questions with images on Stack Overflow. As
shown in the figure, android accounts for the largest numberof questions that have images. However, in terms of proportion,
the questions labeled with the css tag most frequently containimages in the top-10, accounting for 20.3%.
III. (RQ1) W HAT ARE THE CHARACTERISTICS OF IMAGESUSED IN S TACK OVERFLOW QUESTIONS ?
A. Approach

B. Results

To understand the characteristics of images on Stack
Overflow, we conducted a qualitative study of a statistically
representative sample of all developer questions that contain atleast one image in our dataset. Since images may come fromdifferent sources and contain different types of content, we
adopt three dimensions of (a) the image source, (b) the image
content, and (c) the purpose served by the image, which issimilar to prior work [15, 16]. Furthermore, to inform the tool
design on whether support for images is crucial, we analyze
the relationship between the image and the comprehension of
the question understanding. Hence, we in addition manuallyclassify the essentialness of the images. We now describe theconstruction of the representative sample and manual coding.
Representative sample construction. As the entire dataset
is too large to manually examine image characteristics, we draw
a statistically representative sample. The required sample size
was calculated so that our conclusions about the ratio of images
with a specific characteristic would generalize to all developerquestions containing images with a confidence level of 95%
and a confidence interval of 5. The calculation of statisticallysignificant sample sizes based on population size, confidence

Source, Content, and Purpose. Four categories, six categories, and three categories are classified for the image source,
content, and purpose through qualitative analysis, respectively.
For those instances that cannot fit the constructed categories,
we regard them as others including the images that are notable to be accessible with 404.
Table II shows the descriptions along with real examples in
the form of short URLs and the frequency results of the image
characteristics studied. For the image source, we find thatdesktop application is the most common source, accounting for
47%. The next frequent source is web browser, with 26% of theimages being classified. User-created and mobile applications
are less common (11% and 10%, separately). For the image
content, our analysis shows that images most frequently present
a user interface, accounting for 58% of the classified images,
followed by Diagram (11%). The contents of source code andconfiguration are relatively less common, accounting for 5%
and 2%, respectively. For the image purpose, we find that
2 https://www.surveysystem.com/sscalc.htm

3TABLE II
T HE FREQUENCY OF IMAGE SOURCES , CONTENTS , AND PURPOSES (RQ1).

Category

Description/Example

Frequency

SourceDesktop application
Web browser
User-created
Mobile application

shorturl.at/jnoHS
shorturl.at/iqrGM
shorturl.at/doBGR
shorturl.at/iqrGM

180 (47%)
99 (26%)42 (11%)40 (10%)Content

User interface
DiagramResultError code
Source code
Configurationsuch as buttons or an input screen, e.g., shorturl.at/rsDPS
such as designs or charts, e.g., shorturl.at/ksvIQ
such as output from a program, e.g., shorturl.at/fiwMN
such as a warning or error log, e.g., shorturl.at/auzCU
such as code snippets, e.g., shorturl.at/gjzJTsuch as build or dependency files, e.g., shorturl.at/kFGSU

224 (58%)
43 (11%)38 (10%)33 (9%)
19 (5%)
6 (2%)

Undesired output

an image presents an output that a developer would like to
avoid, e.g., shorturl.at/arFX1
an image is used to further support understanding of thedeveloper question, e.g., shorturl.at/akyU
an image show an output that a developer would like to achieve,e.g., shorturl.at/cgmsF

174 (45%)

the question is not understandable without images.
the question is understandable without images

263 (68%)
101 (26%)

PurposeAdditional context
Desired output

Essentialness

Essential
Not essential

the majority of images (45%) are used to provide undesired
output (showing an output that the developer would like to
avoid), while 13% of images are used to provide desired output(showing an output that a developer would like to achieve).
The rest of the images are served to enable additional context(image is used for further understanding of the developer post),
i.e., 37%. In addition, with respect to the essentialness, weobserve that 68% of images are essential to the question context,
which means that these questions cannot be understood without
the images. Such results indicate that the information conveyedfrom the images may fulfill the comprehension needs of thequestions.

User interface

Source code
ResultOthersError code
DiagramConfiguration

141 (37%)
48 (13%)

Undesired output
OthersDesired output
Context understanding

Fig. 2. Relationship between Image Contents and Purposes (RQ1).

Relationship between image characteristics. Furthermore,
we investigate the relationship between image content andpurposes by drawing a parallel category diagram. Parallel setsare variants of parallel coordinates, in which the width oflines that connect sets corresponds to the frequency of theirco-occurrence. Figure 2 depicts the relationship between image
contents and purposes. We observe that different image contents
are likely to show the primary purposes. For example, in termsof image contents User interface, Result, and Error code,
images tend to present undesired outputs (52%, 53%, and 97%,
respectively). Although in terms of image contents Source code,
Diagram, and Configuration, images are more likely to support
context understanding (89%, 58%, and 100%, respectively).

content view looks ugly, see attachment image", the image is
obviously attached to present undesired output. The author
used the word ‘ugly’ to depict the output. If no image isprovided, the developer who attempts to answer this question
would have no idea how ugly it is. Therefore, it is essential
for the user to attach images in this case. Figure 3 (b) shows
an example of a question in which the image is taken from a
mobile virtual machine. In this case, we classify its source as
mobile application. The mobile screen displays an interface
that includes an input box; therefore, it is labeled as the userinterface. As the question describes, the developer shares thisimage trying to complement the situation where the image
shown can be properly placed in android studio while
it cannot work in the specific mobile model (i.e., Samsung
Galaxy s3). Based on this, we classify the purpose of theimage as additional context. In this instance, the image isconsidered essential since it is tricky for a developer to imagine
the vision of the transparent buttons and the proper layout.

Representative Examples. Now we show two representative
examples to illustrate the categories of the source, content, andpurpose of the images. Figure 3 (a) shows an example inwhich the image is captured from a desktop application called
TYPO3. TYPO3 is a Web content management system. We
classify the image content into user interface since the menu
list and function icons of TYPO3 are clearly displayed. As
the text written in the question body is “But the backend

4IV. (RQ2) H OW ARE IMAGES USED IN QUESTIONS ?
A. Approach
To understand what types of questions are likely to include
images, we perform a qualitative analysis to classify 384
representative samples that are used in RQ1. For the typesof questions, we refer to the coding schema provided by apioneering work by Treude et al. [11]. The coding schema
consists of the following nine categories:• how-to: Questions that ask for instructions.
• discrepancy: Some unexpected behavior that the person
asking the question wants explained.
• environment: Questions that ask the environment either
during development or after deployment.
• error: Questions that include a specific error message.
• decision help: Questions that ask for an opinion.
• conceptual: Questions that are abstract and do not have aconcrete use case.
• review: Questions that implicitly or explicitly ask for acode review.
• non-functional: Questions about non-functional requirements such as performance or memory usage.
• novice: Often explicitly states that the person asking thequestion is a novice.
• noise: Questions not related to programming.
Similarly to RQ1, we then conducted our manual coding inseveral rounds following a systematic method. Since the coding
book already existed, two authors participated in this manual
analysis. We first selected 30 questions from our representative
samples and the first two authors independently classified
them into nine types of questions, in order to validate thecomprehensive understanding of the coding schema. However,
the measured fixed-marginal Kappa score was not sufficientin the first round. Then they opened discussions and tried toreach a consensus on disagreements between them. During
these discussions, the third author, who has more than ten yearsof Stack Overflow research experience, joined as a referee toresolve each disagreement. In the second round, the first two
authors coded another 30 samples and the fixed-marginal Kappascore reached 0.90, indicating almost perfect. Inspired by thisresult, the remaining representative samples (i.e., 324 questions)were equally divided into two sets, and the first two authors
independently coded them, respectively.
Control Group. In addition, we construct a control group
to fairly compare the difference in terms of question types
between questions with images and those without images. To
do so, we randomly selected an equal number of 384 questions
without images. Since the measured agreement between thefirst two authors was already almost perfect, similarly, we
divided 384 questions without images into two sets (i.e., 192
for each set). The first author independently coded the first set
and the second author independently coded the second set.
After we classified the types of sampled questions withimages and the control group, we further analyzed the question
characteristics, especially the difference in (I) the description
length and (II) the experience of the questioner between them.

(a) Visual issue report that provides an undesired output.3

xml
android

editText
editText(b) Visual issue report that provides additional information.4
Fig. 3. Two purpose kinds of attaching visual contents.

RQ1 Summary: Our results show that desktop application is the most common image source (47%), followedby the web browser (26%). The images are more likely
to contain the content of the user interface (58%), while
the contents of source code and configuration are theleast common (5% and 2%, respectively). We identify
three types of purposes for sharing images in questions,with the undesired output being the most common(45%). Furthermore, we observe that 68% of images
are essential to the question context, and the questions
cannot be understood without these images.

2 https://stackoverflow.com/questions/21459190/
3 https://stackoverflow.com/questions/36866090/

5

novicenon-functional
novicereview

novicenon-functional
novicereview

discrepancy

discrepancy
User interface

decision help
conceptual

Undesired output

decision help
conceptual

how-toSource code
ResultOthersError code

how-toerror
environment

DiagramConfiguration

error
environment

OthersDesired output

Context understanding(a) Question Types and Image Contents(b) Question Types and Image Purposes.

Fig. 4. Relations between Question Types, Image Contents, and Image Purposes (RQ2)

Kuramoto et al. [19] observed that issue reports with images B. Results
are described in fewer words than non-visual issue reports.
Question Types. Table III shows the frequency of types ofInspired by their work, we assume that images could alsoquestionsthat contain images. As we can see, the most common
reduce the number of words to generate questions. Existingtypeofquestion
is how-to (questions that ask for instructions),
work pointed out that experienced users tend to generate betteraccountingfor46%.
The following frequent type is discrepancy,
questions to maintain their reputation [2]. We conjecture that
with
24%
of
instancesbeing classified. Although these two mostimage as a strategy would be more often used by experienced
frequent
types
areconsistent
with questions without images,
questioners, as they may better summarize the key information
we
find
that
the
proportionofdiscrepancy is relatively higher
from the image contents. Therefore, we introduce the following(24%
versus
18%
as
shown
in
Table
III). This finding suggests
two related hypotheses to test:
that users are more likely to upload images with the intention❏ (H1) “A question with images includes fewer words.”of receiving explanations for some unexpected behavior. On the❏ (H2) “A question with images is more likely to be submitted other hand, we observe that the third common category error isby an experienced user.”less likely to contain images (10% versus 14%). One possible
reason is that, compared to other visual images (such as icons
Note that to calculate the description length of a question, we in the user interface), the structure of the error message is
specifically count the number of words in the body description, organized and could be easier to be replaced using plain text.
excluding its title. Meanwhile, URLs to attach images, code In addition, excluding how-to, discrepancy, and error question
blocks, and tables are not counted as words. The experience types, images are rarely used in the other six types of questionsof the questioner refers to the number of questions that were (environment,conceptual, novice, decision help, non-functional,
submitted before the studied question with images, similar to and review), with a frequency of less than 5%.
the prior work [20]. To validate the statistical significance ofRelationship between image characteristics and question
the two proposed hypotheses, we elect to adopt Bonferroni’s
types. Similar to RQ1, we explore the relationships between
multiple comparison test [21] with α= 0.05 considering multiple
question types and image contents, question types, and image
pairwise comparisons, It is a common method that can be used
purposes. Figure 4 presents these relationships via parallel
to compare different groups at the baseline [22].
category diagrams, specifically where (a) shows the relationshipbetween question types and image contents, and (b) describes
the relationship between question types and image purposes.
Figure 4 (a) intuitively shows that most of the image contents
TABLE III
used in how-to and discrepancy questions are the user interface,
T HE FREQUENCY OF QUESTION TYPES THAT CONTAIN IMAGES (RQ2).
with 70% and 56%, respectively. From Figure 4 (b), we findthat images contained in discrepancy and error questions tend
Category
With image Without image
to display undesired output, i.e., 60% and 67%, respectively.
how-to185 (46%)
162 (42%)
Question Characteristics. Figure 5 presents the results
discrepancy
93 (24%)69 (18%)of the comparison of the characteristics of the questions:error
41 (10%)55 (14%)
environment
13 (3%)
16 (4%)
specifically (a) for the description length and (b) for the
Question Types conceptual
13 (3%)
10 (3%)
experience of the questioner. As shown in Figure 5 (a), no big
novice13 (3%)
11 (3%)
differences (close median scores) are observed for the mostdecision help
7 (2%)
25 (7%)
non-functional
8 (2%)
10 (3%)
frequent question types (i.e., how-to, discrepancy, and error).
review
7 (1%)
14 (4%)
It may indicate that the image does not reduce the number of6

Question-Typewords to describe the questions. Furthermore, the statistical
test scores α are greater than 0.05 for the comparison of all
types (referred to All) and the frequent ones, suggesting that
there is no significant difference. On the other hand, as shownin Figure 5 (b), we visually see that there exists a difference interms of comparison of the experience of the questioner. For
example, within the category All, the median of Without Imageis 4 while the median of With Image is 11. Similar findings
can be observed for the most common type of questions (howto). Furthermore, the statistical test confirms the significant
difference, with α being smaller than 0.05. We now summarize
the validation of the two proposed hypotheses:❏ (H1) “A question with images includes fewer words.” is notsupported.
❏ (H2) “A question with images is more likely to be submitted
by an experienced user.” is supported.

noviceconceptual
decision help
discrepancy
environment
error
how-tonon-functional
review
All

Without Image
With Image

0

100

200

Description Length

300

400

(a) Comparison of Description Length.

Question-TypeRQ2 Summary: Our results show that how-to, discrepancy, and error are the three most common question
types to contain images. Among them, discrepancy
questions are relatively more frequent compared tothe questions without images with an increase of 6%.
Furthermore, our statistical test shows that there isno significant difference in description length between
questions with images and the ones without images,
but images tend to be used by experienced users.

noviceconceptual
decision help
discrepancy
environment
error
how-tonon-functional
review
All

Without Image
With Image

0

20

40

60

80

100

Questioner Experience

120

140

160

(b) Comparison of Questioner Experience.

V. (RQ3) H OW ARE QUESTIONS WITH IMAGES ANSWERED ?
A. Approach

Fig. 5. Comparison of Question Characteristics (RQ2)

In this research question, we aim to investigate the roleof images in receiving answers. Specifically, we perform aquantitative method to analyze the difference between thequestions with images and the ones without images in termsof the following three metrics:• (I) Has Answers: denotes if not a question receives ananswer. Asaduzzaman et al. [10] reported that the numberof unanswered questions has increased significantly. Theirwork also revealed that unanswered questions are usually
not clear and too difficult to understand. In this sense, we
assume that images could help to improve the answer rate
as an image is worth a thousand words.
• (II) Accepted Answers: represents whether or not aquestion receives an accepted answer. Recent work statedthat as of August 2019, 47% of Stack Overflow questions
were unresolved [6]. If developers do not get theiraccepted answers, the websites will not be useful. Priorwork [8] also claimed that code snippet is an importantfactor related to the question quality. Thus, similarly, wehypothesize that images may help questioners get acceptedanswers compared to those with only plain text.
• (III) Answer Time: refers to the period between the timeof submission of the question and the first answer timein this study. Wang et al. [23] argued that the user
experience when asking questions on Q&A websites canbe significantly improved by reducing the needed time to
get an answer. Since the image serves as complementary
information, we speculate that it will help the questioners
receive an answer faster. Note that in the computation of
the time period, we exclude those questions that do notcontain any answers.
Based on the three metrics and their rationales, we further
propose the following three hypotheses to test:❏ (H3) “A question with images receives more answers thanthose without images.”❏ (H4) “A question with images receives more acceptedanswers than those without images.”❏ (H5) “A question with images takes less time to receive thefirst answers.”
For the (H3) and (H4), we elect to invoke the Chi-square test,
a nonparametric hypothesis test that is largely used to compare
two independent proportions (i.e., between the question withimage and the question without image). For (H5), similar
to RQ2, we apply Bonferroni’s multiple comparison test as
we have multiple categories of question types (e.g., how-to,
discrepancy, etc).

7

TABLE IV
C OMPARISON OF A NSWERED Q UESTIONS (RQ3)
Accepted Answer

how-todiscrepancy
error
decision help
noviceconceptual
environment
non-functional
review
Total

Without image

With images

Without image

110 (60%)
45 (48%)21 (51%)6 (86%)
5 (45%)
9 (69%)
6 (46%)
3 (50%)
4 (67%)
205 (56%)78 (48%)32 (46%)21 (38%)14 (56%)5 (45%)
3 (30%)
10 (63%)4 (40%)
8 (57%)
175 (47%)

158 (88%)
75 (82%)30 (77%)6 (86%)
10 (91%)10 (83%)
11 (85%)5 (83%)
6 (100%)311 (85%)

147 (91%)
57 (82%)45 (82%)23 (92%)10 (91%)8 (80%)
16 (100%)
9 (90%)
13 (93%)328 (88%)

there is no statistically significant difference. This indicates thatimages may not speed up the time to get an answer. However,
we cannot explain the causality in the current stage, as acceptedanswers and time could be affected by multiple confounding
factors. In the end, we summarize the validation of the three
proposed hypotheses:❏ (H3) “A question with images receives more answers thanthose without images” is not supported.
❏ (H4) “A question with images receives more acceptedanswers than those without images” is supported.
❏ (H5) “A question with images takes less time to receive thefirst answers.” is not supported.

B. Results

Question-Type

Answer Characteristics. Table IV presents the comparison
results in terms of Has Answers and Accepted Answers andFigure 6 shows the comparison of Answer Time.
First, as shown in Table IV, we observe that the presenceof images does not greatly help to increase the probability ofgetting answers. The proportion of answered questions evenslightly decreases for the two common question types (i.e.,
how-to and error). Furthermore, the Chi-square test confirmsthat there is no significant difference. Second, when we inspect
the proportion of accepted answers, we find that, in general,questions with images are more likely to get accepted answers.
For example, the proportion of accepted answers for questions
with images is 56%, while the proportion of accepted answers
for questions without images is 47%. With a closer look at
the types of questions, the same findings are observed within
the several types including the most popular type (how-to,
60% against 48%). The statistical tests validate that there is asignificant difference in the proportion of accepted answers with
α < 0.05. However, for the second common type (discrepancy
questions), no significant difference is found between the twogroups. Third, on the contrary, no obvious difference is seen in
Figure 6 in terms of the answer time regardless of All or thefrequent question types. Bonferroni’s test further confirms that

noviceconceptual
decision help
discrepancy
environment
error
how-tonon-functional
reviewAll

RQ3 Summary: The results show that questions
with images are more likely to get accepted answers,
especially the common question types (i.e., how-to anderror). However, the images do not significantly speed
up the time to receive the first answer compared tothose without images.
VI. I MPLICATIONS AND C HALLENGES
Based on our results for the three research questions, we now
discuss the implications and challenges for both developers
and researchers.
Classifying images by content, purpose, and sources provides
richer information to complement a Stack Overflow question.
Results from RQ1 reveal six types of image contents (i.e.,
User interface, Source code, Error code, Diagram, Results, andConfiguration) and four types of image sources (i.e., Desktop
Application, Web Browser, Mobile Application, and Usercreated). Furthermore, we observe that images are essential
and provide complementary information compared to theirassociated text. As shown in Table 2, we find that 68% ofimages are essential for the questions. For example, in thisquestion5 , the developer faced a problem when using Bootstrap
input groups and applying the negative margin. To seek
answers, the developer posted an image showing the unexpecteduser interface where the bottom highlight of the input was
hidden to support the problem description. Complementing

Without Image
With Image

0

20

40

60

80

AnswerTime

Has Answers

With image

100

120

140

Fig. 6. Comparison of Answer Time (RQ3).

5 https://stackoverflow.com/questions/21631763/

8the work of Nayebi [1] who reported that 87.8% of survey
respondents agreed that images provide additional informationcompared to text, we empirically show how images are usedand how essential they are to understand a question. For
developers, our results provide concrete recommendations on
the types of images that are commonly shared in specificsituations. Stack Overflow in general intends to be against
the usage of images as raised in this post6 , hence our study
would be used to complement the evidence-based guidelines.
For example, developers are commonly required to take ascreenshot of their desktop (47%) to present a user interface(58%). Furthermore, users tend to provide images when asking
about undesired outputs in their user interface. Meanwhile, in
the aforementioned developer post, the developer expressed
one of the major concerns of “links to images fail”. But ourclassification results may also indicate that the failure cases(questions with 404 images) are relatively less frequent. For
researchers, the automatic classifiers can be proposed based on
our constructed taxonomy of image characteristics for a largescale study. For example, to analyze the effect of confounding
factors in future work, statistical models require quite large data,
where a reliable classifier can be employed to automaticallylabel the factors related to image features (such as contents
and purposes).
Specific questions with images would have a higher chance
to get accepted answers. In RQ3, our results show that asignificant difference in the proportion of accepted answers
can be found between the questions with images and those
without images. For example, the probabilities of how-to, error,
and decision help questions to receive accepted answers reach
60%, 51%, and 86%, respectively, which is much higher
than those with images. Combined with the observation in
RQ2, where despite the presence of images, the description
length does not decrease significantly, it may indicate thatthese types of questions with images would be more clear
and ease comprehension. Therefore, we recommend that users
include images and explain them properly when submitting
the above specific question types. Prior work [2] reported thatcode snippets are important for the quality of a question, but
not that important for the speed of getting an answer. Similarly,
though images are an important element, in our work, there areno significant differences in terms of the answer time. The next
logical step would take into account other confounding factors
and see the role of images in the statistical models. Anotherpossible step is to conduct a developer survey to understandhow they regard questions with images. We hypothesize thatimages may potentially increase cognitive load.
Effect of images on the Stack Overflow downstream task.
During the manual classification process in RQ1, we additionally annotated to what extent the images contain text. The
results show that almost 93% of the images studied contain text.
We conjuncture that images, which are regarded as one of themost efficient communication methods in modern society, thesetexts along with image features would be beneficial for thedownstream work. Especially today, the technology of computer
vision develops dramatically. For example, one of the state-ofthe-art OCR (Optical Character Recognition) techniques named
paddleocr7 has played an important role in both academia
and industry. Two prospective downstream tasks are foreseen.
For example, automatic question generation has attracted
significant interest in the Stack Overflow community [9, 24].
The latest work specifically proposed an approach (TaskKG4Q)
to generate how-to questions [4]. Another useful task isconcerning question relatedness in order to relieve inefficientand time-consuming question search [25, 26]. We notice thatthese downstream works have not yet taken the informationfrom the visuals (such as image contents, purposes, and text
in the images). Thus, in future work, it is essential to evaluate
whether visuals can facilitate these automatic tasks.
VII. R ELATED W ORK
In this section, we position our work with respect to theliterature on Stack Overflow and non-textual information
sharing in software development.
A. Stack Overflow
Stack Overflow is a gold mine for software engineeringresearch and has been found to be useful for softwaredevelopment. Regarding knowledge extraction from Stack
Overflow, Treude et al. [11] categorized the types of questions
asked and explored which questions are answered well andwhich remain unanswered. Wang et al. [27] used Latent
Dirichlet Allocation (LDA) topic modeling to categorise Stack
Overflow questions so that they can automatically categorise
new questions. Nasehi et al. [28] identified characteristics
of effective examples by analyzing well-received answers.
Similarly, Treude and Robillard [29] conducted a survey-based
study to understand developers’ information needs as they relate
to code fragments. Stack Overflow is also used to understandand address challenges in new domains. Wan et al. [30] used
Stack Overflow to understand the challenges and needs among
blockchain developers by applying Balanced LDA. Abdellatif etal. [31] examined Stack Overflow to provide insights on topics
that chatbot developers are interested in and the challenges
they face. Yang et al. [32] conducted a large-scale study on
Stack Overflow to identify security-related questions asked bypractitioners and investigated the popularity and difficulty ofdifferent topics. Apart from a large body of existing empiricalwork, many tools have been proposed to assist developers
with the usage of Stack Overflow. Treude and Robillard [33]presented SISE, a novel machine learning-based approach, toextract insight sentences from Stack Overflow and use themto improve API documentation. Gao et al. [34] proposed anovel question generation task based on a sequence-to-sequencelearning approach to formulating high-quality question titles
from given code snippets. Most recently, Nayebi [1] reported
a steady increase in sharing images over the past five years in
Stack Overflow. Their developer survey showed that images are

6 https://meta.stackoverflow.com/q/285551

7 https://github.com/PaddlePaddle/PaddleOCR

9meaningful and provide complementary information comparedto their associated text.
Although the textual contents and the code snippets of Stack
Overflow are widely analyzed, the characteristics of images
that are shared in questions remain unclear. Thus, in this study,
we conduct an empirical study to explore image characteristics
and investigate their role in questions, in order to complement
the literature knowledge in terms of the question quality.
B. Non-Textual Information Sharing in Software Development
Modern development tools are often integrated with or supplemented by communication channels and social media, to support developers’ collaboration and communication needs [35].
Specifically, developers tend to share links, images, or videos tofulfill information needs during software development. Several
studies have shown that link sharing plays a significant role.
Hata et al. [15] investigated the characteristics of links in sourcecode comments, the purpose of these links, and how they evolveand decay, using GitHub as a data source. Their results showthat links are prevalent in source code repositories and thatlicenses, software homepages, and specifications are common
types of link targets. Specifically, links are rarely updated,but many link targets evolve. In the context of code review
settings, Wang et al. [36] observed seven intentions behind link
sharing, and the results of their developer survey suggest thatlink sharing is useful. Meanwhile, Hirao et al. [37] categorized
five types of review linkage, such as patch dependency, broader
context, and alternative solution, and suggested that review
linkages could improve reviewer recommendation. In addition
to link sharing, developers are increasingly making use of
screenshots and screen-recordings as a means of reporting
problems [38–40]. Wang et al. [36] proposed SETU which
combines information from screenshots and textual descriptions
to detect duplicate crowdtesting reports. The evaluation results
showed that SETU can significantly outperform existing stateof-the-art approaches. Cooper et al. [39] introduced TANGO
to help developers determine whether video-based bug reports
represent the same bug. TANGO combines tailored computer
vision techniques, optical character recognition, and text
retrieval and could reduce developer effort by more than
60%. The latest work [41] has successfully extracted relevant
information automatically from gameplay videos to detect bugsin video games and shows promising evaluation results.
Similarly to previous studies, our work serves as the firststep towards understanding the role of shared images in Stack
Overflow that play in fulfilling information needs. Our work
would further encourage future studies on the effect of images
on their application and also complements other research thatlooks into the usage of alternative media for communications
among developers (e.g., visuals, links, etc.).
VIII. T HREATS TO VALIDITY
We now discuss the threats to the validity of our study.
External Validity. External Validity is concerned with ourability to generalize based on our results. We only conduct
an empirical study on Stack Overflow. As such, observations

10

based on this study may not generalize to other Q&A forums.
However, Stack Overflow is nowadays regarded as the mostpopular Q&A forum involved with a large number of questions
and developers. On the other hand, our sample of images
is diverse, including images related to varying programming
languages. Another threat may be related to the time span of thestudied SOTorrent datasets (as of December 2019). However,
we believe that our proposed research questions (characteristics
of images, questions, and answers) are not largely affectedby the timing factor. However, replication studies may help
improve the strength of generalizations that can be drawn.
Construct Validity. Construct validity refers to the relation
between theory and observation. We summarize three potential
threats to construct validity. First, in the identification ofimages that are shared in the questions, we rely on a listof file extension types of images. However, cases might occur
where some images are not retrieved by any type in this list.
Nevertheless, to mitigate this threat, we make sure that we cover
the most common extension types, i.e., JPG, PNG, GIF, WEBP,
TIFF, PSD, RAW, BMP, HEIE, INDD, JPEG, SVG, AI, EPS,
and PDF. Second, when classifying the image characteristics,
we only take into account the first image appearing in thequestion. However, a question may contain more than one
image. We believe that it may not have a large impact on
our findings in this paper since one question tends to focus
on one specific topic. Third, in our qualitative analysis ofimage characteristics (i.e., source, content, purpose), and themanual annotation of question types may be miscoded due tothe subjective nature of our coding approach. To mitigate thisthreat, similar to prior work [36, 42], we took a systematic
approach to test our comprehension with 30 continuous samples
using Kappa agreement scores from three or two individuals(RQ1 and RQ2, separately). If the scores are not acceptable,
we continue to have a discussion to refine and use anothergroup of samples to evaluate until the Kappa score indicatesthat the agreement is substantial (0.61–0.80) or almost perfect(0.81–1.00). Afterward, we were able to complete the codingof the rest of the sample dataset independently.
Internal Validity. Internal validity is the approximate truth
about inferences about cause-effect or causal relationships.
Two related threats are summarized. The first threat is related
to the metrics that we selected and measured to understandthe role of images. For example, when we calculated theexperience of users, the historical questions before the visualquestions studied may also contain images, which would affectour final conclusion. Meanwhile, apart from the presence ofimages, several confounding factors may have an effect on
the answer time, but our goal in this stage is to shed light on
the role of images. The second threat concerns the choice ofelected statistical tests. To validate our five proposed hypotheses,
we adopted different hypothesis tests (Bonferroni’s multiple
comparison test and chi-square test) taking into account thecharacteristics of the target data (whether independent or not).
The conclusion of significance may differ from the statisticaltests; however, we are confident since these two tests arebroadly used in empirical studies.

IX. C ONCLUSION AND F UTURE W ORK
In this paper, we perform an empirical study on Stack Overflow, to understand the role of images. Specifically, we propose
three research questions to investigate the characteristics of
the image, the use of images in questions, and the role ofimages in receiving answers. Our RQ1 results primarily showthat various images are shared in Stack Overflow with theundesired output being the most frequent purpose, and images
are essential to understanding many questions. RQ2 results
show that how-to, discrepancy, and error are the three mostcommon types of questions that contain images, but there areno significant differences in description length. RQ3 results
show that questions with images are more likely to get acceptedanswers than those without images, but images do not shorten
the time to receive answers.
In summary, our study highlights the role that images playin Stack Overflow questions. Specifically, the image is served
as an important non-textual resource to complement the users’
information needs to better understand the question context.
The next logical step would be a deeper study of investigating
the causality between images and answers, and understanding
whether images would increase the cognitive load by a user
survey. Future research direction also includes studying therole of the images in answers, and the effect of images on
downstream tasks by injecting image-related attributes.
X. ACKNOWLEDGMENTS
This work has been supported by JSPS KAKENHI Grant
Numbers (JP20H05706, JP21H04877, JP22K18630) and JST
PRESTO Grant Number JPMJPR22P6.
R EFERENCES[1] M. Nayebi, “Eye of the mind: Image processing forsocial coding,” in Proceedings of the 42nd International
Conference on Software Engineering: New Ideas andEmerging Results, 2020, p. 49–52.
[2] L. Ponzanelli, A. Mocci, A. Bacchelli, M. Lanza, andD. Fullerton, “Improving low quality stack overflow post
detection,” in Proceedings of the 30th IEEE international
conference on software maintenance and evolution, 2014,
pp. 541–544.
[3] M. M. Rahman, J. Barson, S. Paul, J. Kayani, F. A. Lois,
S. F. Quezada, C. Parnin, K. T. Stolee, and B. Ray, “Evaluating how developers use general-purpose web-search forcode retrieval,” in Proceedings of the 15th International
Conference on Mining Software Repositories, 2018, pp.
465–475.
[4] M. Liu, X. Peng, A. Marcus, C. Treude, J. Xie, H. Xu, andY. Yang, “How to formulate specific how-to questions insoftware development?” in Proceedings of the 30th ACM
Joint European Software Engineering Conference andSymposium on the Foundations of Software Engineering,
2022, pp. 306–318.
[5] D. Correa and A. Sureka, “Chaff from the wheat:
Characterization and modeling of deleted questions on

11

stack overflow,” in Proceedings of the 23rd international
conference on World wide web, 2014, pp. 631–642.
[6] M. Yazdaninia, D. Lo, and A. Sami, “Characterizationand prediction of questions without accepted answers on
stack overflow,” in Proceedings of the 29th International
Conference on Program Comprehension, 2021, pp. 59–70.
[7] M. Duijn, A. Kucera, and A. Bacchelli, “Quality questions
need quality code: Classifying code fragments on stack
overflow,” in Proceedings of the 12th Working Conferenceon Mining Software Repositories, 2015, pp. 410–413.
[8] F. Calefato, F. Lanubile, M. C. Marasciulo, and
N. Novielli, “Mining successful answers in stack overflow,”in Proceedings of the 12th Working Conference on Mining
Software Repositories, 2015, pp. 430–433.
[9] Z. Gao, X. Xia, J. Grundy, D. Lo, and Y.-F. Li, “Generating question titles for stack overflow from mined codesnippets,” ACM Transactions on Software Engineeringand Methodology, vol. 29, no. 4, pp. 1–37, 2020.
[10] M. Asaduzzaman, A. S. Mashiyat, C. K. Roy, andK. A. Schneider, “Answering questions about unansweredquestions of stack overflow,” in Proceedings of the 10th
Working Conference on Mining Software Repositories,
2013, pp. 97–100.
[11] C. Treude, O. Barzilay, and M.-A. Storey, “How doprogrammers ask and answer questions on the web?
(nier track),” in Proceedings of the 33rd International
Conference on Software Engineering, 2011, p. 804–807.
[12] F. Calefato, F. Lanubile, and N. Novielli, “How to ask
for technical help? evidence-based guidelines for writing
questions on stack overflow,” Information and Software
Technology, vol. 94, pp. 186–207, 2018.
[13] Replication package. [Online]. Available: https://anonymous.4open.science/r/SOImages/[14] S. Baltes, L. Dumani, C. Treude, and S. Diehl, “Sotorrent:
Reconstructing and analyzing the evolution of stack
overflow posts,” in Proceedings of the 15th International
Conference on Mining Software Repositories, 2018, p.
319–330.
[15] H. Hata, C. Treude, R. G. Kula, and T. Ishio, “9.6
million links in source code comments: Purpose, evolution,and decay,” in Proceedings of the 41st International
Conference on Software Engineering, 2019, pp. 1211–
1221.
[16] T. Xiao, D. Wang, S. Mcintosh, H. Hata, R. G. Kula,
T. Ishio, and K. Matsumoto, “Characterizing and mitigating self-admitted technical debt in build systems,” IEEE
Transactions on Software Engineering, pp. 1–1, 2021.
[17] R. V. Krejcie and D. W. Morgan, “Determining sample
size for research activities,” Educational and Psychological Measurement, vol. 30, no. 3, pp. 607–610, 1970.
[18] A. J. Viera, J. M. Garrett et al., “Understanding Interobserver Agreement: The Kappa Statistic,” Family Medicine,vol. 37, no. 5, pp. 360–363, 2005.
[19] H. Kuramoto, M. Kondo, Y. Kashiwa, Y. Ishimoto,K. Shindo, Y. Kamei, and N. Ubayashi, “Do visual issue
reports help developers fix bugs?” in Proceedings of the30th International Conference on Program Comprehension, 2022, pp. 511–515.
[20] D. Abric, O. E. Clark, M. Caminiti, K. Gallaba, and
S. McIntosh, “Can duplicate questions on stack overflow
benefit the software development community?” in Proceedings of the 16th International Conference on Mining
Software Repositories, 2019, pp. 230–234.
[21] R. J. Simes, “An improved bonferroni procedure formultiple tests of significance,” Biometrika, vol. 73, no. 3,
pp. 751–754, 1986.
[22] S. Lee and D. K. Lee, “What is the proper way toapply the multiple comparison test?” Korean journal ofanesthesiology, vol. 71, no. 5, pp. 353–360, 2018.
[23] S. Wang, T.-H. Chen, and A. E. Hassan, “Understanding
the factors for fast answers in technical q&a websites,”
Empirical Software Engineering, vol. 23, no. 3, pp. 1552–
1593, 2018.
[24] N. Duan, D. Tang, P. Chen, and M. Zhou, “Question
generation for question answering,” in Proceedings ofthe 2017 conference on empirical methods in natural
language processing, 2017, pp. 866–874.
[25] B. Xu, A. Shirani, D. Lo, and M. A. Alipour, “Predictionof relatedness in stack overflow: deep learning vs. svm:
a reproducibility study,” in Proceedings of the 12th International Symposium on Empirical Software Engineeringand Measurement, 2018, pp. 1–10.
[26] J. Pei, Y. Wu, Z. Qin, Y. Cong, and J. Guan, “Attentionbased model for predicting question relatedness on stack
overflow,” in Proceedings of the 18th International
Conference on Mining Software Repositories, 2021, pp.
97–107.
[27] S. Wang, D. Lo, and L. Jiang, “An empirical study on
developer interactions in stackoverflow,” in Proceedings ofthe 28th Annual ACM Symposium on Applied Computing,2013, p. 1019–1024.
[28] S. M. Nasehi, J. Sillito, F. Maurer, and C. Burns, “Whatmakes a good code example?: A study of programming
q amp;a in stackoverflow,” in Proceedings of the 28th
IEEE International Conference on Software Maintenance,
2012, pp. 25–34.
[29] C. Treude and M. P. Robillard, “Understanding stack
overflow code fragments,” in Proceedings of the 33rd
International Conference on Software Maintenance andEvolution, 2017, pp. 509–513.
[30] Z. Wan, X. Xia, and A. E. Hassan, “What do programmers
discuss about blockchain? a case study on the use ofbalanced lda and the reference architecture of a domain
to capture online discussions about blockchain platforms
across stack exchange communities,” IEEE Transactions
on Software Engineering, pp. 1331–1349, 2021.
[31] A. Abdellatif, D. Costa, K. Badran, R. Abdalkareem,and E. Shihab, “Challenges in chatbot development:

12

A study of stack overflow posts,” in Proceedings ofthe 17th International Conference on Mining Software
Repositories, 2020, pp. 174–185.
[32] X.-L. Yang, D. Lo, X. Xia, Z.-Y. Wan, and J.-L. Sun,
“What security questions do developers ask? a large-scale
study of stack overflow posts,” Journal of Computer
Science and Technology, pp. 910–924, 2016.
[33] C. Treude and M. P. Robillard, “Augmenting api documentation with insights from stack overflow,” in Proceedingsof the 38th International Conference on Software Engineering, 2016, p. 392–403.
[34] Z. Gao, X. Xia, J. Grundy, D. Lo, and Y.-F. Li, “Generating question titles for stack overflow from mined codesnippets,” ACM Transactions on Software Engineeringand Methodology, vol. 29, no. 4, pp. 1–37, 2020.
[35] M. Chui, J. Manyika, and J. Bughin, “The social economy: Unlocking value and productivity through social
technologies,” McKinsey Global Institute, Tech. Rep.,
2012.
[36] D. Wang, T. Xiao, P. Thongtanunam, R. G. Kula, andK. Matsumoto, “Understanding shared links and theirintentions to meet information needs in modern code
review,” Empirical Software Engineering, vol. 26, no. 5,
p. 96, 2021.
[37] T. Hirao, S. McIntosh, A. Ihara, and K. Matsumoto, “The
Review Linkage Graph for Code Review Analytics: A
Recovery Approach and Empirical Study,” in Proceedingsof the 26th International Symposium on the Foundationsof Software Engineering, 2019, p. 578–589.
[38] L. MacLeod, A. Bergen, and M.-A. Storey, “Documenting and sharing software knowledge using screencasts,”
Empirical Software Engineering, vol. 22, pp. 1478–1507,
2017.
[39] N. Cooper, C. Bernal-Cárdenas, O. Chaparro, K. Moran,and D. Poshyvanyk, “A replication package for it takes
two to tango: Combining visual and textual information
for detecting duplicate video-based bug reports,” inProceedings of the 43rd International Conference on
Software Engineering: Companion Proceedings, 2021,
pp. 160–161.
[40] J. Wang, M. Li, S. Wang, T. Menzies, and Q. Wang,“Images don’t lie: Duplicate crowdtesting reports detection
with screenshot information,” Information and Software
Technology, vol. 110, pp. 139–155, 2019.
[41] M. R. Taesiri, F. Macklon, and C.-P. Bezemer, “Clip meets
gamephysics: Towards bug identification in gameplay
videos using zero-shot transfer learning,” in Proceedingsof the 19th International Conference on Mining Software
Repositories, 2022, pp. 270–281.
[42] D. Wang, R. G. Kula, T. Ishio, and K. Matsumoto,“Automatic patch linkage detection in code review using
textual content and file location features,” Informationand Software Technology, vol. 139, p. 106637, 2021.