Detecting Functional Differences using Automatic
Test Generation for Automated Assessment in
Programming Education
Ryoko Izuta∗ , Shinsuke Matsumoto∗ , Hiroshi Igaki† , Sachio Saiki‡ , Naoki Fukuyasu† and Shinji Kusumoto∗∗ Osaka University, Osaka, Japan

Email: {r-izuta, shinsuke, kusumoto}@ist.osaka-u.ac.jp† Osaka Institute of Technology, Osaka, Japan
Email: {hiroshi.igaki, naoki.fukuyasu}@oit.ac.jp‡ Kochi University of Technology, Kochi, Japan
Email: saiki.sachio@kochi-tech.ac.jp

Abstract—Software testing is being leveraged in programming
education for automated assessment of programming assignments. When using software testing in programming education,
program specifications are provided as unit or integration tests,
and students create programs that pass these tests. Although
this method has various advantages, such as ensuring objective
program specifications and automating the operation check, italso has many disadvantages. For example, detecting innovations,such as original specifications and functional extensions by anindividual student, is difficult. The purpose of this researchis to automatically detect functional differences among student
programs in programming education using tests. In our proposed
method, automatic test generation is applied to student programs,
and the generated tests are mutually executed for other student
programs. Furthermore, we classify the tests based on theexecution path to obtain sets of tests that are capable of detecting
functional differences.
Index Terms—automated assessment, automated test generation, programming education, test execution path

I. I NTRODUCTION
Many researches propose using software testing in programming education (hereinafter referred to as test-based
education) [1]–[4]. In test based education, the specificationsof a class assignment can be explicitly defined as a unit
test or an integration test (hereinafter referred to as pre-test).
Providing pre-tests has various advantages to both students andinstructors, such as ensuring objective task specifications andautomating the task completion check. Test based education
also has a high affinity with modern and practical software development such as test-driven development [5] and continuous
integration [6]. In addition to the acquisition of programmingskills, it is expected to gain practical experience in softwaredevelopment.
A problem with test-based education is that it is difficult
to detect integuity made by individual students. Integuity isa functional difference between student source codes. This
includes functional improvements and extensions that students
implement in order to achieve a better grade. A typicalexample of integuity is the process of checking the validityof parameters, such as checking whether an object is null orchecking the range of an integer value. Furthermore, original
additional specifications and functional extensions that are notchecked by the pre-tests are also considerable. So, integuity
can be said as implementations outside the scope of pretesting. We believe that teachers should detect integuity, give
positive feedback and encourage their efforts, rather than
forcing everyone to implement minimal functions.
However, it is fundamentally impossible to test all possible
behaviors that exist [7]. Excessive testing, while effective interms of software quality, is not an adequate strategy foreducation because it not only obscures the essence of the task
specification, but also inhibits free thinking in programming.
We believe that the implementation of humor that contributes
to the enjoyment of programming, such as easter eggs, should
be detected and praised in some way, rather than ruthlessly
rejected by pre-tests. Moreover, other researches that focus
on automatic assessment, such as AUTOGRADER [8], only
assess functions that instructors expect and cannot detect
integuity.
The goal of this research is the automatic detection of
integuity among student source code in test-based education.
In order to achieve this goal, we set the following challenges,
as shown in Fig. 1.
• Challenge 1: How to generate tests with the ability todetect integuity and classify these tests.
• Challenge 2: How to automatically detect integuity using
the generated tests.
In Challenge 1, we generate tests from each source codedeveloped by students that have the ability to detect integuity,which cannot be achieved by the pre-tests alone. In Challenge
2, the resulting tests are applied to each source code to detect
the presence of integuity.
In this paper, we propose a test clustering method that uses
automatic test generation and execution path information toachieve Challenge 1. In this method, we first apply automatic
test generation to all submitted source code to obtain tests withthe ability to detect integuity. Furthermore, every generated test
is applied to all of the students, and the execution paths areminimal
implementation

includeincludepositive integuity negative integuity...

Challenge 1: Generate and classify tests

1
2
3
4
5
6
7
8
9

public PortNumber {int number;public PortNumber(int n) {number = n;}public getNumber() {return number;}public toString() {if (number == 22) return "ssh";
if (number == 80) return "http";... }}(a) Example code for PortNumber classTest* if only pre-tests

1
2
3
4

@Test public void test1() {int number = new PortNumber(22).getNumber();
assertEquals(22, number);}(b) Pre-test for PortNumber constructor
Challenge 2: Automatically detect integuity

Fig. 2: Example code and pre-test for PortNumber class
1
2
3

Fig. 1: Two challenges in automatic detection of functional
differences

measured in order to extract the steps to verify the behaviorof each test. Finally, by vectorizing and clustering the paths ofeach test, we identify the tests that have the same verification
procedure. In this paper, we also report the results of apreliminary experiment on a programming subject having a
few lines of code.
II. M OTIVATING E XAMPLE
In this section, we explain the motivation of this researchusing simple java source code. The subject is the PortNumber
class, which represents port numbers in the TCP/IP protocol.
An example code of this class and a pre-test for the constructor
are shown in Fig. 2. This PortNumber class consists ofthe following elements: an int type field number (line 2)
representing the port number, the constructor (line 3), a gettermethod for number (line 4), and a stringification method fornumber (lines 5 through 8).
As the first implementation task of this class, students are
told to write their own PortNumber class constructor thatpasses the pre-test shown in Fig. 2(b). Two possible answers
are shown in Fig. 3. The constructor of Student A (ConstructorA) in Fig. 3(a) is exactly the same as the example constructor
and implements only the minimum functionality that passes
the pre-test. On the other hand, Constructor B in Fig. 3(b)implements a validation check of the port number and does
not accept negative values.
Constructor B is an implementation that contains positive
features not included in the example, which should be detected
and graded appropriately. However, the pre-test only checks
whether the constructor implements the minimal function
correctly, and does not have the ability to check additional
functions. Although visual confirmation is an option, it is notrealistic considering the number of students in programming

public PortNumber(int n) {number = n;}(a) Constructor of Student A
1
2
3
4

public PortNumber(int n) {if (n < 0) number = 0; // originality
else number = n;}(b) Constructor of Student B

Fig. 3: Student constructor examples

courses, such as university courses which could include 10 to
100 students.
III. P ROPOSED M ETHOD
A. Overview
In order to detect integuity in the source code of multiple
students, we use automatic test generation. First, tests are
generated automatically from each student source code. Then,each test is applied to every student, not just the base student
used for test generation, to identify functional differences.
Simply applying automated test generation will result innumerous tests. These tests most likely include tests that can
detect functional differences. However, many redundant tests
are also included, such as tests that only check the samefunctionality as the pre-tests. Visual inspection is necessary inorder to understand the “meaning” of a test and the functional
difference that is represented by that test. In order to reduce
the effort required, appropriate classification of the generated
tests is needed.
Therefore, the proposed method is divided into three major
steps, as follows: (1) automatic test generation to generatetests with the ability to detect integuity, (2) classification of
the generated tests, and (3) visual inspection of the functional
differences. In this paper, we explain Steps 1 and 2. Step 3 isrelated to Challenge 2 and is left for a future study.

TABLE I: Subject of the experiment

B. Step 1. Automated Test Generation
First, we apply automatic test generation to every student’s
source code and generate unit tests. There are many studies
on automatic test generation [9], and open-source tools, suchas EvoSuite [10] and Randoop [11] , have been released.
Automatic test generation uses exploratory meta-algorithms,
such as genetic algorithms, and execution path analysis togenerate tests that maximize coverage.
Functional differences between source codes often appear
as differences in branches, i.e., execution paths. Therefore,
tests with the ability to detect integuity can be obtained byautomated test generation, which aims to maximize coverage.
C. Step 2. Test Classification
In Step 2, we first mutually apply the tests generated
in Step 1 to every student’s source code. If a total of M
tests are generated from N source codes, then N × M test
executions are performed. In addition, for each test execution,
the execution path is recorded. The mutual execution of tests
and the recording of path information are the preprocessing
steps for the following test classification.
Here, we consider the criteria that should be used to
calculate the similarity of tests in the test classification phase.
The goal of this research is to automatically detect integuity.
Therefore, the similarity should be based on the function thateach test is testing.
An option is to use the state of each test execution (pass/fail). However, test states are not suitable for classifying thesetests because test states are automatically generated. Since
the true oracle of the tests are unknown [12], automatedtest generation assumes that the base source code behavescorrectly. As a result, there is excessive variety in the test
assertions, and some assertions might even be in contradiction.
We decompose a test into two parts: the execution part,which represents the procedure of the test, and the assertion
part, which checks whether the execution result matches the
expected value. The execution path of only the execution partbest represents what function each test is testing. Therefore, werecord the execution path and use it to classify the generated
tests. Unlike other researches about test prioritization [13]–[15], by using the path information, classification is possible
from the viewpoint of the check procedure of each test, leaving
out the assertion part.
In order to use the path information, we calculate a vector
for each test. The component of each vector is a hash of aserialization of the student ID of the class executed and theexecution path. For example, if student source code x is tested
and the execution path is 11101 (meaning that only the fourthline out of all five lines was not passed), then string x11101
is generated and hashed. Note that each vector component is
a hash value and cannot be treated as a numeral.
Finally, the tests are classified based on the aforementioned
vectors. Currently, tests that have the exact same vectors areclassified as belonging to the same set. In this way, tests
that have the same execution paths, i.e., tests with the samefunctions, will be classified together. This test classification

Name of subject

When n < 0When n > 65535

CheckNone
substitute n
substitute n
CheckLow-SetAltV
substitute 0 *a
substitute n
CheckLow-ThrowE
throw exception*a
substitute n
CheckUpp-SetAltV
substitute n
substitute 0*b
CheckUpp-ThrowE
substitute n
throw exception*b*a
CheckBoth-SetAltV
substitute 0
substitute 0*b*aCheckBoth-ThrowE
throw exception
throw exception*b*a Integuity 1 (lower bound check), *b Integuity 2 (upper bound check)

method can be fully automated even for source codes with
100 participants, and can classify a large number of generated
tests in terms of the similarity of their verification procedures.
IV. P RELIMINARY EXPERIMENTS
We applied our approach to a programming subject having a
few lines of code in order to confirm whether our approach cangenerate tests with the ability to detect integuity and classify
them appropriately.
A. Subjects
The subject used in the experiments is the PortNumber classas described in Section II. The assumed integuity consists ofa combination of two factors. One is whether or not to checkthe range of the port number, and the other is the behavior incase of an error. There are three types of range confirmation:
lower limit confirmation (n < 0), upper limit confirmation(n > 65535), and both lower limit confirmation and upper
limit confirmation. There are two types of behavior when thespecified port number is out of range: assigning an alternate
value (zero) and raising an exception.
As a result of the combination of the above two factors,
we have created a subject with seven functional differences asshown in Table I. The names of the subjects are a combinationof range checking and error handling. For example, CheckNone does not perform any range checking and assigns the
specified value as it is. This is the minimum implementationof the issue requirements and corresponds to the constructor
shown in Fig. 3(a). In CheckLow-SetAltV, if the value is below
the lower limit, then an alternative value is assigned. No upper
limit check is performed. This is equivalent to the constructor
shown in Fig. 3(b).
The integuity assumed in this preliminary experiment iswhether the two types of range checking processes are performed, and the behavior in case of an error should be ignored.
For example, the bottom four lines of Table I (CheckUpp-* andCheckBoth-*) should all be detected as having the same functional difference (checking the upper bound), although they
behave differently in case of an anomaly. For the definition ofthis functional difference, please refer to Section III-C.
B. Experimental Method
First, we manually created seven types of subject sourcecode as described in Section IV-A. Second, we used EvoSuite,Student source ChkNone ChkLow- ChkLow- ChkUpp- ChkUpp- ChkBoth- ChkBoth- Set
Test
SetAltV ThrowE SetAltV ThrowE SetAltV ThrowE ID
ChkNone1

765d

cc29

4e8e

a0e6

4f09

cc48

e794

1

ChkNone2

765d

cc29

4e8e

a0e6

4f09

cc48

e794

1

ChkNone3

765d

8daa

c8a0

a0e6

4f09

214f

e04b

2

ChkLow-SetAltV1

765d

cc29

4e8e

a0e6

4f09

cc48

e794

1

ChkLow-SetAltV2

765d

8daa

c8a0

a0e6

4f09

214f

e04b

2

ChkLow-SetAltV3

765d

cc29

4e8e

a0e6

4f09

cc48

e794

1

ChkLow-ThrowE1

765d

cc29

4e8e

a0e6

4f09

cc48

e794

1

ChkLow-ThrowE2

258a

9a26

c8a0

e831

e087

54db

e04b

3

ChkLow-ThrowE3

765d

cc29

4e8e

a0e6

4f09

cc48

e794

1

ChkUpp-SetAltV1

765d

cc29

4e8e

a0e6

4f09

cc48

e794

1

ChkUpp-SetAltV2

765d

8daa

c8a0

a0e6

4f09

214f

e04b

2

ChkUpp-SetAltV3

765d

cc29

4e8e

849f

7e7a

dc18

08f1

4

ChkUpp-SetAltV4

765d

cc29

4e8e

a0e6

4f09

cc48

e794

1

ChkUpp-ThrowE1

765d

cc29

4e8e

a0e6

4f09

cc48

e794

1

ChkUpp-ThrowE2

765d

8daa

c8a0

a0e6

4f09

214f

e04b

2

ChkUpp-ThrowE3

258a

55aa

f528

2da1

7e7a

25a9

08f1

5

ChkUpp-ThrowE4

765d

cc29

4e8e

a0e6

4f09

cc48

e794

1

ChkBoth-SetAltV1 765d

cc29

4e8e

a0e6

4f09

cc48

e794

1

ChkBoth-SetAltV2 765d

cc29

4e8e

a0e6

4f09

cc48

e794

1

ChkBoth-SetAltV3 765d

cc29

4e8e

849f

7e7a

dc18

08f1

4

ChkBoth-SetAltV4 765d

8daa

c8a0

a0e6

4f09

214f

e04b

2

ChkBoth-SetAltV5 765d

cc29

4e8e

a0e6

4f09

cc48

e794

1

ChkBoth-ThrowE1 765d

cc29

4e8e

a0e6

4f09

cc48

e794

1

ChkBoth-ThrowE2 765d

cc29

4e8e

a0e6

4f09

cc48

e794

1

ChkBoth-ThrowE3 258a

55aa

f528

2da1

7e7a

25a9

08f1

5

ChkBoth-ThrowE4 258a

9a26

c8a0

e831

e087

54db

e04b

3

ChkBoth-ThrowE5 765d

cc29

4e8e

a0e6

4f09

cc48

e794

1

# generated tests

3

3

3

4

4

5

5

# path types

2

4

3

4

3

5

3

Fig. 4: List of generated tests and runtime paths

an automatic test generation tool, with the default parameters
to generate tests for the subject source code. Third, we ran
every test for every source code along with JaCoCo1 , anexecution path measurement tool, to obtain path information
during the test execution. Finally, we calculated the vector
for each test with the method explained in Section III-C. The
md5sum command was used for hashing.
C. Results and Discussion
The hashes for each test obtained from the experiment are
shown in Fig. 4. Each column represents a source code, andeach row represents a test method. Each value represents thepath hash for the execution. Each test vector consists of thehashes on its right. In order to improve visibility, the samehash in each column is filled with the same color. The nameof each test method is represented with the name of thesource code from which the test was generated followed bya sequential number. Note that the order of the tests in thevertical direction is based on the order generated by EvoSuiteand has no meaning. The set IDs in the right-hand columnof the table are the classification results based on the exact
matching of the vectors. Furthermore, the meaning of each set
is shown in Table II, and examples of a specific generated test
for sets ID1 to ID3 are shown in Fig. 5.
As a result of clustering, most of the tests were assigned
to set ID1. There is a variety of input values n, but we cansee from the test meaning in Table II and the test example
in Fig. 5(a) that all of the tests in set ID1 have inputs in
1 https://www.jacoco.org/jacoco/

1
2
3
4
5

@Test public void CheckNone2() {PortNumber portNumber0 = new PortNumber(992);
int int0 = portNumber0.getNumber();
assertEquals(992, int0);}(a) Example of set ID1
1
2
3
4
5

@Test public void CheckNone3() {PortNumber portNumber0 = new PortNumber(-1);
int int0 = portNumber0.getNumber();
assertEquals(-1, int0);}(b) Example of set ID2
1
2
3
4
5
6
7
8
9

@Test public void CheckLowThrowException2() {PortNumber portNumber0 = null;try {portNumber0 = new PortNumber(-394);
fail("Expecting exception:
IllegalArgumentException");} catch(IllegalArgumentException e) {verifyException("PortNumber", e);}}(c) Example of set ID3

Fig. 5: Examples of sets ID1 through ID3

the normal range (0 ≤ n ≤ 65535). These set ID1 tests are
generated from all of the source codes and can be interpreted
as a shared function.
Tests in set ID2, such as Fig. 5(b), are tests that check
what happens with a negative input. Both tests that expect thenegative value itself and tests that expect an alternative value
zero were classified as belonging in set ID2. This shows thattests checking the lower bound are properly classified to thesame set without depending on the assertion part. All tests in
set ID4 likewise check the upper bound.
On the other hand, sets ID3 and ID5 were sets overly
subdivided. Set ID3 is identical to set ID2 in terms of testing n≤ 0, and ID5 is identical to ID4 in terms of testing n ≥ 65535.
These sets should be classified as the same set. This excessive
division occurs because of unexpected path deviation. As
shown in Fig. 5(c), all tests expect thrown executions andlack the call to the getter method. The presence or absenceof the getter call affects the execution path, which results ina difference in the path vectors and the classification. Indeed,
CheckNone takes two different paths, 765d and 258a, as
shown in Fig. 4 even though CheckNone implements only theminimal function. Tests with 765d include getter calls, whiletests with 258a do not. Only the tests with 258a belong tosets ID3 and ID5. Tests that expect thrown exceptions tend tobe overly subdivided, and this needs to be addressed.
V. C ONCLUSION
We proposed a test classification method that uses automatic test generation and execution path information to detect
integuity among student source code.
In the future, we plan to address the following issues. First,
it is essential to deal with the excessive test segmentation

TABLE II: Set ID of the test and its meaning
set ID

Range of n to substituteexpectation

1
2
3
4
5

0 ≤ n ≤ 65535
n<0
n<065535 < n65535 < nnn or alternative value 0
Exceptionn or alternative value 0
Exceptionthat occurred in the preliminary experiment. In addition, theautomatic detection of integuity using generative tests, which
we mentioned as Challenge 2, is an important remaining task.
In order to achieve Challenge 2, we are considering clustering
based on test state in addition to the current clustering, andanalyzing the characteristics of each group (number of test
passes, number of tests in the same group, etc.) that emerge
from the results. The application of the system to educationalsettings is also an important issue, and we are considering
introducing the system to the programming exercises that weare conducting.
ACKNOWLEDGMENT
This research was partially supported by JSPS KAKENHI
Japan (Grant Numbers: JP21H04877, JP17K00500,
JP19K02973, JP19K03001, and JP21K11829)
R EFERENCES[1] F. Restrepo-Calle, J. J. Ramı́rez Echeverry, and F. A. González, “Continuous assessment in a computer programming course supported bya software tool,” Computer Applications in Engineering Education,
vol. 27, no. 1, pp. 80–89, 2019.
[2] P. Ihantola, T. Ahoniemi, V. Karavirta, and O. Seppälä, “Review ofrecent systems for automatic assessment of programming assignments,”
in Proc. Koli Calling International Conference on Computing Education
Research, 2010, pp. 86–93.
[3] S. H. Edwards and M. A. Pérez-Quiñones, “Experiences using testdriven development with an automated grader,” Journal of Computing
Sciences in Colleges, vol. 22, no. 3, pp. 44–50, 2007.
[4] C. Douce, D. Livingstone, and J. Orwell, “Automatic test-based assessment of programming: A review,” J. Educ. Resour. Comput., vol. 5,
no. 3, pp. 4—-es, 2005.
[5] C. Desai, D. Janzen, and K. Savage, “A survey of evidence for testdriven development in academia,” SIGCSE Bulletin, vol. 40, no. 2, pp.
97–101, 2008.
[6] B. P. Eddy, N. Wilde, N. A. Cooper, B. Mishra, V. S. Gamboa, K. M.
Shah, A. M. Deleon, and N. A. Shields, “A pilot study on introducing continuous integration and delivery into undergraduate software
engineering courses,” in Proc. Conference on Software Engineering
Education and Training, 2017, pp. 47–56.
[7] C. Kaner, C. Cem, and K. All, “The impossibility of complete testing,”
1997.
[8] X. Liu, S. Wang, P. Wang, and D. Wu, “Automatic grading of
programming assignments: An approach based on formal semantics,”
in Proc. International Conference on Software Engineering: Software
Engineering Education and Training, 2019, pp. 126–137.
[9] S. Anand, E. K. Burke, T. Y. Chen, J. Clark, M. B. Cohen,
W. Grieskamp, M. Harman, M. J. Harrold, P. McMinn, A. Bertolino,
J. Jenny Li, and H. Zhu, “An orchestrated survey of methodologies
for automated software test case generation,” Journal of Systems andSoftware, vol. 86, no. 8, pp. 1978–2001, 2013.
[10] G. Fraser and A. Arcuri, “Evosuite: automatic test suite generationfor object-oriented software,” in Proc. ACM SIGSOFT Symposium andEuropean Conference on Foundations of Software Engineering, 2011,
pp. 416–419.

[11] C. Pacheco and M. D. Ernst, “Randoop: Feedback-directed random
testing for java,” in Proc. ACM SIGPLAN Conference on ObjectOriented Programming Systems and Applications Companion, 2007, pp.
815–816.
[12] F. Pastore, L. Mariani, and G. Fraser, “Crowdoracles: Can the crowd
solve the oracle problem?” in Proc. International Conference on Software Testing, Verification and Validation, 2013, pp. 342–351.
[13] M. Khatibsyarbini, M. A. Isa, D. N. Jawawi, and R. Tumeng, “Test caseprioritization approaches in regression testing: A systematic literature
review,” Information and Software Technology, vol. 93, pp. 74–93, 2018.
[14] J. Chen, L. Zhu, T. Y. Chen, D. Towey, F.-C. Kuo, R. Huang, and Y. Guo,
“Test case prioritization for object-oriented software: An adaptive random sequence approach based on clustering,” Journal of Systems andSoftware, vol. 135, pp. 107–125, 2018.
[15] R. Carlson, H. Do, and A. Denton, “A clustering approach to improving
test case prioritization: An industrial case study,” in Proc. IEEE International Conference on Software Maintenance, 2011, pp. 382—-391.