Is Cutting-Edge Software Engineering Attractive forDevelopers in SMEs?
Yukasa Murakami, Masateru Tsunoda
Graduate School of Science and Engineering Research
Kindai University
Higashiosaka, Japanm.yukasa@gmail.com, tsunoda@info.kindai.ac.jp

Abstract—To set the goal of software engineering research,
we should aware needs of the majority of developers. In Japan,
the majority is developers who belong to companies where
number of employees is smaller than 300. However, it is not clear
to what extent cutting-edge software engineering satisfies theneeds of the majority. So, we analyzed whether cutting-edge
software engineering satisfies the needs of developers who belong
to small and medium-sized enterprises, or not. We asked 16
software developers who belong to software development
companies about effectiveness of cutting-edge researches to theirwork. In the analysis, we stratified data based on the size of theircompany. The analysis results suggested that when size ofcompanies to which subjects belong was different, research
categories related to subjects’ work were also different. However,sizes of companies did not affect evaluation to the effectivenessand the interests of the researches. Additionally, we predictedcategories of researches which relate to each developer’s business(i.e., discrimination), and the evaluation of developers (evaluatedby five grades) to each research, using collaborative filtering. As
a result, when we predicted research categories which related tosubjects’ work, the accuracy was low (F1 score was 27%). In
contrast, when we predicted the interestingness to the researches,
the accuracy was moderate (Average of absolute error was 1.16).
Keywords—investigation;
developers; recommendationquestionnaire;

softwareI. INTRODUCTION
Recently, software is one of infrastructures of society, andembedded software and enterprise software are indispensable
for social activities. Therefore, high reliability is required forsoftware. On the other hand, large software is required to be
developed in short term. To enhance the probability of successof such projects, assistances based on software engineering
techniques are needed.
When we cope with studies of the assistances based on
software engineering, we should be aware of needs of majorityof software developers. In Japan, a software development
company is legally regarded as a small and medium-sizedenterprise (SME), when the number of employees is smaller
than 300 or stated capital is smaller than 300 million yen [15].
The ratio of software development companies where number of
employees is smaller than 300 is 99% in Japan. Additionally,
the ratio of software developers who are employed by thecompanies is 69% [8]. Therefore, the majority of softwaredevelopers in Japan belong to such companies with less than
300 employees. There are also many SMEs in the UnitedStates of America [16] and EU [4].
It is not clear to what extent cutting-edge software
engineering satisfies the needs of the majority of softwaredevelopers. Some studies investigated the effectiveness orcorrectness of direction of software engineering forpractitioners [9][12]. However, subjects of their investigation
were developers who belong to large companies such as
Facebook or Microsoft. In small and medium-sized companies,
circumstances such as number of software project members
and size of software are considered to be different from large
companies. So, when size of a company is different, needs ofdevelopers may be also different.
In this study, we analyze whether cutting-edge software
engineering satisfies the needs of developers who belong tosmall and medium-sized enterprises, or not. It is important for
cutting-edge software engineering to satisfy the needs of large
companies. Similarly, cutting-edge software engineering
should consider the needs of small and medium-sizedenterprises. In our study, subjects who belong to softwaredevelopment companies evaluated the effectiveness of cuttingedge software engineering for their business. After that, we
stratified the data based on size of their companies, andanalyzed it. In addition, we predicted categories of researches
which relate to each developer’s business, and the evaluationof developers to each research, using collaborative filtering.
The purpose of the prediction is to support the evaluation ofresearch by developers.
II. RELATED WORK
Impact Project [10] analyzed the influence of software
engineering research to software development. For example,
they analyzed how software engineering contributes toevolution of programming language and software configuration
managements [3][13]. As a result, they concluded that software
engineering research affects actual software development
process. The result suggests effectiveness of software
engineering. However, it is not clear that whether cutting-edge
software engineering is also effective or not, and ratio ofeffective cutting-edge studies (when some studies are noteffective).

TABLE I.
Number of employees
Less than 300
300 to 999
More than 999

SUMMERY OF SUBJECTS

Number of subjects Years of experience < 3
Average years of experience
Master's degree
6
4
6.7
2
5
5
1.2
2
5
5
1.2
1

TABLE II.
Number of employees
Less than 300
Less than 300 (Years of exp. < 3)
300 to 999
More than 999

DEVELOPMENT PROCESS IN WHICH SUBJECTS ARE INVOLVED.
Requirement
Basic
Detailed
Development
Project
Testing Coding
analysisdesign design
supportmanagement
33%
67%
67%
100%
83%
33%
17%
0%
50%
50%
100%
75%
0%
0%
20%
60%
60%
80%
80%
20%
0%
20%
60%
60%
100%
80%
0%
20%

Rubin et al. [12] discussed directions of software
engineering research. They interviewed developers who belong
to frontier software companies such as Google and Facebook.
They clarified issues faced by the developers, and based on that,they suggest needs for researches which help developers toaccess useful information quickly. However, they do notclarify needs of developers who do not belong to frontier
software companies.
There are two studies which are similar to our study [2][9].
The studies were conducted by the same research group. In thestudies, subjects who belong to software development
companies read summaries (about one or two lines) of papersof international conferences on software engineering. After that,
subjects evaluated value of the researches by five grades. As aresult, about 70% of studies are regarded as valuable.
Our investigation procedure is based on the existing studies[2][9]. However, our study has three major differences fromthem. The first one is viewpoint of evaluation to each research.
In this study, subjects evaluated researches based on usefulness
on their own businesses. This is the most important difference.
The second one is that our analysis focused on size of software
companies to which subjects belong. In the existing studies,
major subjects belong to large software companies such as
Microsoft. The third one is that researches were evaluatedbased on concise explanatory documents (They are longer than
the two-line abstracts). As a result, the evaluation to cuttingedge studies was not high, in contrast to the existing studies.
III. DATA COLLECTION
We collected data from developers who belong to software
companies, using a questionnaire (Google Forms). Subjects
answered questions such as effectiveness of researches basedon the following procedure:

3.

Subjects evaluate papers based on usefulness for theirwork by five grades (1: lowest and 5: highest). In
addition, they evaluate papers based on their interest.

We decided categories of the researches and assignment of
papers to the categories, based on ICSE (International
Conference on Software Engineering) 2016. ICSE is the top
conference on software engineering field. In more detail, we
set each session in the conference as each category of
researches. The detail of the categories is explained in section
IV.B. Each category (i.e., session) includes about four papers.
To shorten answer time of subjects, subjects read summaries ofresearches [7], instead of reading papers of them directly. Eachsummery consists of two or three presentation slides, and it isrelatively easy to understand.
Additionally, subjects answered the following questions
about themselves: Academic degree (e.g., bachelor or master) Years of work experience Number of employees in subjects’ companies (less than
300, 300 to 999 employees, more than 999)
 Development process in which subjects are involved.
IV. ANALYSIS OF ANSWERS
To clarify the purpose of the analysis, we set the following
research questions.
 RQ1: Do research categories relate to development
work biasedly?
 RQ2: When size of subjects’ company is different,
research categories which relate to their works is also
different?

1.

Choose categories of researches which relate to theirwork (at least three categories, and at most fivecategories).

 RQ3: Is results of cutting-edge researches effective forsubjects’ works?

2.

Read the concise explanatory documents of eachresearch which belongs to the top three categories
selected in step 1.

 RQ4: When size of subjects’ company is different,
effectiveness of researches to their work is also
different?

TABLE III.

RATIO OF RESEARCH CATEGORIES RELATED TO SUBJECTS’ WORK.
# ofpapers

No. Research category
1
2
3
4
5
6
7
8
9
10
11
12
13
15
16
17
18
19
20
21
22
23
24
25
26
27

Android (Software on Android)
Performance (Performance of software)
Empirical (Software data analysis)
Symbolic Execution (Software analysis by symbolic execution)
Compilers and Emerging Trends
Energy and Videos (Energy saving of software, and utilizing videos)Open Source (Open source software)
Defect Prediction (Software defect prediction)
Synthesis (Automatic code generation)
API (Application Programming Interface)
Code Smells (Coding methodologies)
Architecture (Software architecture)
Testing (Software testing)Effort Estimation and Search (Software effort estimation)
Product Lines (Reuse software by product lines)
Repair and Model Synthesis (Automatic software repair)
Languages (Programming languages)
Debugging (Debugging software)
Requirements (Software requirements)
Dynamic Analysis (Dynamic analysis of software)
Security
Collaborative (Human factor of software development)
Software Quality
Program Analysis
Concurrency (Concurrent execution of software)
Maintenance (Software maintenance)

A. Overview of Subjects
Subjects of this study is 16 software developers who belong
to software development companies. Table I is summary of
subjects, stratified by size of their companies. On the groupwhere number of employees was smaller than 300 (i.e., firstrow of the table), there were two developers whose years ofwork experience was more than three years. On other groups,
years of work experience was less than three years. To
suppress the influence of years of work experience, we also
show the analysis results, excluding subjects whose years ofwork experience was more than three years from the analysis.
Development process of subjects: Table II showsdevelopment process in which subjects are involved, stratified
by size of their companies. Most of subjects were involved incoding and testing. About half of subjects were involved indesign. Subjects were not involved in requirement analysis anddevelopment support, when company size was small and yearsof work experience was low (i.e., second row in the table). In
other cases, some subjects were involved in the process.
B. Research Categories Relating to Business
Table III shows research categories used in the analysis.
We used sessions of ICSE conference as research categories.
We added explanations to each categories as shown in theparenthesis. There were two testing sections in the conference,
and we merged the sessions in the analysis. Each session
included about four research papers.
Analysis of all subjects (relating to RQ1): Table III also
shows research categories which were regarded to relate

4
4
3
4
4
4
2
4
4
3
4
3
8
3
4
4
4
4
4
4
4
4
3
4
4
4

Total

Less
than
300

Less than300
300 to(Years of 999
exp. < 3)

2%
4%

4%
4%
4%
2%
4%
20%
2%

11%

8%

11%

8%

More
than 999

7%

8%
8%

7%

8%
8%

6%
17%
6%

17%
8%

4%
9%
7%
2%
2%
4%
9%

6%
11%
6%

8%
8%
8%

6%

13%

17%

7%
21%

23%

7%
7%
7%
7%

8%
8%

8%

7%
7%

8%
15%

25%

14%

8%

6%

subjects’ work in subjects’ answers. In the table, “total”
column indicates ratio of answers which recognized therelationship without stratification by company size. As
explained in section 3, each subjects select categories, andtherefore the denominator is the total number of selectedcategories (e.g., multiply 16 subjects by 3...5 categories). Other
columns denotes the ratio, stratified by company size. Total of
the ratio is 100% on each column.
On 36% (9 out of 26 categories) of the categories, therelationship to subjects’ work were not observed. This isbecause some categories do not relate to ordinary softwaredevelopment very much. For example, “compilers andemerging trends” and “energy and videos” are not considered
to relate to ordinary software development.
Analysis categories such as “empirical” category were notselected by subjects, although some subjects are involved indevelopment support. So, data analysis may not relate to
subjects’ work very much (i.e., low priority) when the numberof employees was smaller than 300 (Note that this group
includes subjects whose years of work experience is more than
three years).
Therefore, the answer to RQ1 is “Yes (There are someresearch categories which do not relate to subjects’ work verymuch).”
Analysis stratified by company size (relating to RQ2):“Performance” and “collaborative” categories did not relate to
subjects’ work explicitly (i.e., the categories were not selectedby subjects), when the number of employees was less than 300.
In contrast, some subjects selected the categories in other cases

(Note that the number of subjects who selected the categories
was four, and therefore it is not ignorable). As shown in Table
II, development process in which subjects are involved was notvery different from each subject. So, the difference of theprocess is not considered to affect the result very much. On thegroup where number of employees was smaller than 300, there
were developers whose years of work experience was morethan three years. That is, it is not probable that the lack of theexperience affected the result. We think that the result arosefrom the differences of characteristics of target software andproject organization (e.g., number of developers in a project)between small companies and other companies.
“API” and “open source” categories related to subjects’
work, when the number of employees was less than 300. In
other cases, the strength of the relationships were weak (i.e.,the categories were not selected by subjects). The categories
were selected by subjects belonging to small companies,
regardless of their years of work experience. So, the work
experience is not considered to affect the result. Also, theprocess in which subjects are involved is not very different
among subjects, as explained before. When software size isrelatively small (i.e., in small companies), the influence of API
and OSS to software may be relatively large, and this may
affect the analysis results.
The results suggest that when the process in which subjects
are involved is not very different, but size of companies towhich subjects belong is different, research categories related
to subjects’ work may be also different. So, the answer of RQ2
is “Yes.” evaluation to research
C. Evaluation to ResearchThis section explains analysis results relating to RQ3 and
RQ4. Table IV shows evaluation of researches included in eachcategory. They were evaluated based on effectiveness tosubjects’ work and interestingness to subjects separately. We
stratified data based on size of companies to which subjects
belong. Evaluation of the effectiveness was not very different
when size of companies was different. The average ofevaluation is less than 3. So, cutting-edge software engineering
may not be explicitly effective for improvements of subjects’work.
In Table IV, the interestingness for researches was not low,compared with the effectiveness. However, average of theinterestingness was different among groups in the table. We
assumed that the interestingness is different, when subjects
have engaged in study in graduate school. So, we stratified
subjects based on their academic degree, as shown in Table V.
On average, the interestingness was relatively high, whensubjects’ academic degree was master. Therefore, we consider
that the interestingness for researches was not affected by sizeof companies, but by subjects’ academic degree. From theresult, we answer to RQ4 as “No.”
Table VI shows average of effectiveness of each research
category, and Table VII shows average of the interestingnessof each category. We did not stratified the data because thenumber of data point of each category was small in the tables.
In the table, we removed categories in which the number ofdata points was smaller than 2. Bold face in the table indicates

TABLE IV.

EVALUATION OF SUBJECTS STRATIFIED BY SIZE OFCOMPANIES.

Number of employees
Less than 300
Less than 300 (Years of exp. < 3)
300 to 999
More than 999

TABLE V.

Effect
2.2
2.2
2.4
2.4

Interest
3.6
3.1
2.9
2.7

EVALUATION OF SUBJECTS STRATIFIED BY ACADEMIC
DEGREE.

Academic degree
Bachelor and others
MasterEffect2.3
2.3

Interest
3.0
3.5

the average was larger than 3. In Table VI and VII, “No.”refers to categories in Table III. Even when we focused oneach research category, evaluation of the effectiveness was low
in most cases. Only average of the effectiveness of“performance” and “synthesis” was higher than 3.
In the tables, average of the interestingness was larger than
3 on more than half of research categories. Correlation
coefficients between the interestingness and the effectivenesswas 0.47. That is, although the effectiveness relates to theinterestingness to some extent, the effectiveness does notstrongly depend on the interestingness. On “language”
category, both average of the interestingness and theeffectiveness was smaller than 2, and it was the lowest in the
categories. Topics of the researches included in the category
were mainly C language. C language may not relate to
subjects’ work very much.
So, the answer of RQ3 is “It is not very effective, except
for some research categories.”
V. RECOMMENDATION OF RESEARCH CATEGORY
A. Overview of Experiment
Subjects were interested in some research categories. If
recommender system suggests research categories in whichdevelopers are likely to be interested in advance, it will help
developers search attractive researches. For example,
recommender system suggests “Research categories which
relate to your work are testing and maintenance. You may
interested in testing researches.” to a developer. If developers
know attractive researches easily, more developers will know
cutting-edge software engineering, and some of them give
feedback to researches. It is expected to promote reconsidering
the effectiveness of each research.
To clarify whether the recommender system make accurate
recommendation or not, we predicted research categories inwhich developers are interested, and evaluated the prediction
accuracy. Although one of authors proposed similar
recommender system in [1], independent variables used forprediction are different. Study [1] used evaluation of each item
as independent variables, this study uses profile of developers
explained later. In contrast to study [1], research categories arerecommended for developers not to read research papers by our
method.

TABLE VI.
No.
Effect# of answers

1
2.0
2

2
3.0
3

7
2.5
2

TABLE VII.
No.
Interest# of answers

2
4.0
2

7
4.0
2

AVERAGE OF EFFECTIVENESS OF EACH CATEGORY.
9
3.0
2

10
2.5
2

12
2.5
2

13
2.2
7

18
1.5
2

19
2.3
4

20
2.3
3

23
2.5
2

24
2.3
4

27
2.2
6

AVERAGE OF USEFULNESS OF EACH CATEGORY.
9
3.0
2

10
4.5
2

12
3.5
2

In the experiment, the research categories and degree of theinterest to the categories were predicted, using collaborative
filtering. As shown in section IV.B, some categories did not
relate to subjects’ work very much. Therefore, prediction(discrimination) of research categories is needed as the firststep. For example, the prediction is “Research categories inwhich developer A is interested are maintenance and testing.”
As the next step, we predicted the degree of the interest (fivegrades). For instance, the prediction is “Interest of developer A
to maintenance is 2, and the interest to testing is 5.”
In the prediction, developers’ profile was used as
independent variables. That is, years of work experience,development process in which developers are involved, andsize of their companies were used. The development process
and the size of the companies were transformed into dummy
variables, because they are categorical variables. We applied
leave-one-out cross validation to evaluate the prediction
accuracy.
As evaluation criteria of discrimination (i.e., prediction ofresearch categories), we used precision, recall, and F1 score.
They are widely used to evaluate performance of recommendersystem [6]. When the values are large, the prediction isaccurate. To evaluate prediction of the degree of the interest,
we used absolute error. Absolute error is absolute value ofdifference between predicted value and actual value (actual
evaluation to researches by subjects). When the value is small,the prediction is accurate.
B. Collaborative Filtering
Collaborative filtering is used to recommend preferable oruseful items such as books and music [5][11][14]. There are
two major recommendation methods of collaborative filtering.
The first one is user based method. The user based method
assumes “When users are similar, their preference is alsosimilar.” First, using values of independent variables, cosine
similarity is calculated. Next, when the similarity is high, avalue of dependent variable is predicted based on other similarusers. User based method was proposed to recommend news
articles [11]. The other major recommendation methods is item
based method [14]. The method recommends items based on
similarity of items. Although the method calculates cosine
similarity, the direction is different. That is, user based method
compares each rows, and item based method compares eachcolumns.
Collaborative filtering uses m × n matrix as shown in Table
VIII. ui ∈{u1, u2, ..., um} denotes i-th developers. pj ∈{p1,
p2, ..., pj} is profile of developers such as years of the

13
2.7
7

18
1.5
2

19
3.3
4

20
2.7
3

23
3.0
2

24
3.3
4

27
2.7
6

experiments. tk ∈{r1k, r2k, ..., rmk} is evaluation to research
categories.
C. Results of Prediction
Table IX shows prediction accuracy of the research
categories. The accuracy of the user based prediction was
higher than the item based prediction. However, F1 score of
the prediction was 27%, and hence it was very low accuracy.
Prediction accuracy of the degree of the interest is shown in
Table X. User based prediction was also slightly higher.
Average of the absolute error was slightly larger than 1, andtherefore the accuracy was not very high.
From the result, it is not easy to predict research categories
in which developers are interested. Therefore, it may be betterfor developers to select the research categories without
recommender system. When the number of data points isincreased, the prediction accuracy may be improved. In
contrast, the degree of the interest can be predicted bycollaborative filtering to some extent. To predict research
categories and the degree of the interest, the user based
prediction is more proper than item based prediction.
VI. DISCUSSION
Software engineering strongly relates to actual
development field, compared to other engineering areas. So,
research of software engineering generally consider application

TABLE VIII.

DATA MATRIC USED IN COLLABORATIVE FILTERING.
u1
u2
…
ui
…
um

TABLE IX.

p1
v11
v21
…
vi1
…
vm1

…
…
…
…
…
…
…

pj
v1j
v2j
…
vij
…
vmj

tk
r1k
r2k
…
rik
…
rmk

PREDICTION ACCURACY OF RESEARCH CATEGORIES.

Prediction method
User based
Item based

TABLE X.

p2
v12
v22
…
vi2
…
vm2

Precision
30%
20%

Recall24%
22%

F1 score
27%
21%

PREDICTION ACCURACY OF SUBJECTS’ EVALUATION.

Prediction method
User based
Item based

Average of absolute error
1.16
1.23

 When size of companies to which subjects belong wasdifferent, research categories related to subjects’ work
were also different. For example, “API” and OSS” wereselected only when the number of employees were
smaller than 300.

to actual development. On ICSE, theoretical research is not
dominant, compared with ASE and FSE. So, we picked up
researches presented in ICSE.
When research papers are categorized, we should satisfy
the following points:

 Except for “performance” and “synthesis,” cutting-edge
researches were not evaluated as effective to subjects’work.

 The definitions of the categories are correct.
 Classification of the papers based on the categories iscorrect.

 Size of companies did not affect evaluation to theeffectiveness and the interestingness of the researches.
Academic degree may affect the interestingness.

 The number of papers included in each category is notbiased very much.
Instead of using the sessions of the conference, we canmake new categories to classify the papers. However, it is noteasy to classify the papers, and therefore, it is difficult tosatisfy the above points. We checked papers included in thesessions, and the classification based on the session did notseem incompatible. So, we used the session to classify the
papers, although it is a bit rough approach.
Existing studies [2][9] asked correctness of directions ofresearches (i.e., general values of researches), and theresearches were highly evaluated on the view point. In contrast,
we focused on usefulness of research to each subject’s work,
because the usefulness is more important for the subjects than
the general values. Although years of work experience waslong on some subjects, their evaluation to the usefulness was
low. So, we consider that lack of the experience did not affectsuch low evaluation very much. We regard that subjects
understand researches to some extent, because subjects’interest to researches was not low.
Based on the experience of one of us, when years of work
experience is larger than one year, developers have knowledge
to understand usefulness of technologies. So, they can properly
judge usefulness of researches to some extent, when the
researches relate to developers’ work.
However, when years of the experience is low, knowledge ofsome development process in which subjects are involved may
be relatively low, and it could affect the evaluation. Especially,
to evaluate researches about early development phase more
properly, we should add subjects who have more years of theexperience.

 When we predicted research categories which related tosubjects’ work, the accuracy was low. In contrast, when
we predicted the interestingness to the researches, theaccuracy was moderate.
To conclude the analysis, we need more subjects. In thiscase study, the effectiveness was not high, and we do not thinkthat cutting-edge software engineering does not satisfy needsof developers in SMEs (small and medium-sized enterprises).
Our future work is to increase the number of subjects.
Especially, we should recruit subjects whose years of work
experience is long.
ACKNOWLEDGMENT
This research was partially supported by Japan Society for
the Promotion of Science(JSPS) [Grants-in-Aid for Scientific
Research (C) and (A) (No.16K00113 and No.17H00731)].
REFERENCES
[1]

[2]

[3]

[4]

VII. CONCLUSIONSIn this study, we analyzed software developers’ evaluation
to effectiveness of cutting-edge software engineering for theirwork, focusing on size of companies to which the developers
belong. First, the developers who work in software companies
selected three to five research categories which relate to theirwork. Next, they evaluated researches included in the
categories (about four researches were included in eachcategory) by five grades, considering effectiveness of theresearches to their work and interestingness. We analyzed thedata, and observed the followings:
 Relationships of some research categories todevelopers’ work were weak. For example, “compilers
and emerging trends” and “energy and videos” werenote selected by subjects.

[5]

[6]

[7]

[8]
[9]

T. Akinaga, N. Ohsugi, M. Tsunoda, T. Kakimoto, A. Monden and K.
Matsumoto, "Recommendation of software technologies based oncollaborative filtering," In Proc. of Asia-Pacific Software Engineering
Conference (APSEC), pp. 209-214, 2005.
J. Carver, O. Dieste, N. Kraft, D. Lo, and T. Zimmermann, “How
Practitioners Perceive the Relevance of ESEM Research,” In Proc. ofInternational Symposium on Empirical Software Engineering and
Measurement (ESEM), no.56, 10 pages, 2016.
J. Estublier, D. Leblang, A. Hoek, R. Conradi, G. Clemm, W. Tichy, and
E. Wiborg-Weber, “Impact of software engineering research on thepractice of software configuration management,” ACM Transactions on
Software Engineering and Methodology (TOSEM), vol.14, no.4,
pp.383-430, 2015.
European Commission, What is an SME? http://ec.europa.eu/growth/smes/business-friendly-environment/sme-definition_en
D. Goldberg, D. Nichols, B. Oki, and D. Terry, “Using Collaborative
Filtering to Weave an Information Tapestry,” Communications of theACM, vol.35, no.12, pp.61-70, 1992.
J. Herlocker, J. Konstan, L. Terveen, and J. Riedl, “Evaluating
collaborative filtering recommender systems,” ACM Transactions on
Information Systems (TOIS), vol.22, No.1, pp.5-53, 2004.
IPSJ/SIGSE Working Group of Activation of International research
activities, ICSE2016 study meeting,https://sites.google.com/site/
sereadings/icse2016 (in Japanese).
Ministry of Economy, Trade and Industry, Japan, Survey of Selected
Service Industries in 2015，2016 (in Japanese).
D. Lo, N. Nagappan, and T. Zimmermann, “How practitioners perceive
the relevance of software engineering research,” In Proc. of Joint
Meeting on Foundations of Software Engineering (ESEC/FSE), pp.415425, 2015.

[10] L. Osterweil, C. Ghezzi, J. Kramer, and A. Wolf, “Determining the
Impact of Software Engineering Research on Practice,” Computer,vol.41, no.3, pp.39-49, 2008.
[11] P. Resnick, N. Iacovou, M. Suchak, P. Bergstrom, and J. Riedl,“GroupLens: An Open Architecture for Collaborative Filtering ofNetnews,” In Proc. of Conference on Computer Supported Cooperative
Work (CSCW), pp.175-186, 1994.
[12] J. Rubin, and M. Rinard, “The Challenges of Staying Together While
Moving Fast: An Exploratory Study,” In Proc. of International
Conference on Software Engineering (ICSE), pp.982-993, 2016.
[13] B. Ryder, M. Soffa, and M. Burnett, “The impact of software
engineering research on modern programming languages,” ACM
Transactions on Software Engineering and Methodology (TOSEM),
vol.14, no.4, pp.431-477, 2005.

[14] B. Sarwar, G. Karypis, J. Konstan, and J. Riedl, “Item-BasedCollaborative Filtering Recommendation Algorithms,” In Proc. ofInternational World Wide Web Conference (WWW), pp.285-295, 2001.
[15] The Small and Medium Enterprise Agency, Japan, Definition of small
and medium-sized enterprises, http://www.chusho.meti.go.jp/pamflet/g_book/h22/teigi.html (in Japanese).
[16] United States Trade Representative, Small and Medium-Sized
Enterprises,
https://ustr.gov/trade-agreements/free-trade-agreements/trans-pacific-partnership/tpp-chapter-chapter-negotiating-8