JTDog:a Gradle Plugin for Dynamic Test Smell Detection
Masayuki Taniguchi, Shinsuke Matsumoto, Shinji Kusumoto
Graduate School of Information Science and Technology
Osaka University
Osaka, Japan{m-tanigt, shinsuke, kusumoto}@ist.osaka-u.ac.jp

Abstract—The concept of the test smell represents potential
problems with the readability and maintainability of the test code.
Common test smells focus on static aspects of the source code,such as code length and complexity. These are easy to detect anddo not cause problems in terms of test execution. On the other
hand, dynamic smells, which are based on test runtime behavior,lead to misunderstanding of the test results. For example, rotten
green tests give developers the false impression that the test was
passed without any problems, even though the test was poorly
executed. Therefore, we should detect dynamic smells and takecountermeasures as early as possible through the development.
In this paper, we introduce JTDog, a Gradle plugin for dynamic
smell detection. JTDog has high portability due to its integration
into the build tool. We applied JTDog to 150 projects on GitHub
and conﬁrmed that the JTDog plugin has high portability. In
addition, JTDog detected 958 dynamic smells in 55 projects. JTDog is available at https://github.com/kusumotolab/JTDog, andthe demo video is available at https://youtu.be/t374HYMCavI.
Index Terms—software testing, test smell, Gradle plugin, build
tool, dynamic analysis

I. I NTRODUCTION
Test smells represent potential problems with the readability
and maintainability of the test code [1]. The presence oftest smells can lead to poor test quality, and it is desirable
to detect test smells early and eliminate them through code
improvements, such as refactoring. Typical examples of test
smells are excessively long code and duplicate tests [2]. Whilethese smells are also in production code, there are test-speciﬁc
smells that are based on practices inherent in test design. For
example, the use of control statements in the test code canresult in increased test complexity and bugs in the test code
itself [2].
These general test smells focus on static aspects of test code,and their negative effects are limited to static problems such as
maintainability. In this paper, we call these smells static smells.
For instance, a test code that is too long results in difﬁcultyin understanding the code and increased maintenance costs.
However, such a code does not affect the test execution results.
Similarly, test codes containing control statements indicate
potential problems that can lead to test complexity and bugsin the tests themselves, which are not problematic in terms oftest execution.
On the other hand, there are smells that affect the resultsof test execution. We call these dynamic smells. For example,

a rotten green test [3] is a passed test that contains unexecuted assertions. Rotten green tests give developers the false
impression that the test was passed without any problems,
even though veriﬁcation by assertions is insufﬁcient. Dynamic
smells, like the rotten green test, can lead to more serious
problems than static smells. Dynamic smells cause problems
with test execution results and lead to misunderstanding of test
results.
Many studies have been conducted on test smells [4]–[7],and none of these studies consider the classiﬁcation of test
smells (static or dynamic). In addition, many tools have been
proposed to detect test smells, such as TestHound [8] andtsDetect [9], and these tools detect only static smells. To ourknowledge, there is no tool for dynamic smell detection.
In this paper, we introduce JTDog, a Gradle plugin for
detecting dynamic smells. Although static smells can be detected only by static analysis of the test code, dynamic smells
require dynamic analysis and various processes, such as test
execution and coverage analysis of the test code. The userof a dynamic smell detection tool must input a great deal of
information, including source code location and classpath, anda dynamic smell detection tool is difﬁcult to use immediately.
Therefore, the portability of dynamic smell detection tools
tends to be lower than that of static smell detection tools. Asa countermeasure to the problem of reduced portability, we
can achieve high portability by incorporating a dynamic smell
detection technique into the Gradle build tool. JTDog detects
the three major dynamic smells: the rotten green test [3], theﬂaky test [10], and the dependent test [11].
In order to conﬁrm the portability of JTDog, we applied the
proposed plugin to 150 projects on GitHub. As a result, JTDog
worked correctly in 122 projects and detected 958 dynamic
smells in 55 of these projects.
II. JTD OG
JTDog is a Gradle plugin for dynamic test smell detection
for Java projects. JTDog detects three types of dynamic test
smells, i.e., the rotten green test [3], the ﬂaky test [10], andthe dependent test [11], and outputs the results to a JSON ﬁle.
JTDog was developed as open-source software on GitHub1 ,and the JTDog plugin is available on the Gradle Plugin Portal2 .
1 https://github.com/kusumotolab/JTDog
2 https://plugins.gradle.org/plugin/com.github.m-tanigt.jtdog

A. Usage
JTDog provides a task labeled sniff that detects dynamic
smells. Developers can detect dynamic smells present in a
project by simply running the command gradle sniff. Furthermore, we can use this task by adding a single line of usage
declaration to the plugins{} block in the build.gradle ﬁle, as
shown below.
plugins {id ’com.github.m-tanigt.jtdog’ version ’latestVersion’}

B. Design
Dynamic Smell Detection Methods: In this subsection, weshow the detection methods for each of three dynamic smells:
the rotten green test, the ﬂaky test, and the dependent test.
Rotten green test We follow the method of Delplanque etal. [3] to detect rotten green tests. First, we statically analyze
the test code to determine the elements of each method,
including assertions and method calls. Then, we run each test,
and if each test is passed, then we analyze the coverage data to
determine whether the test contains any unexecuted assertions.
Flaky test A ﬂaky test is a test that both passes and fails
periodically with same code. Many methods for detecting ﬂaky
tests have been proposed in recent years. One of the simplest
methods, called RERUN, is to rerun a failed test multiple times
and to check whether the test passes even once [10]．Other
methods include monitoring changes in coverage due to codechanges and classifying failed tests using machine learning[12], [13]．JTDog detects ﬂaky tests by RERUN.
Dependent test A dependent test is a test that depends onother tests. If the test order changes, then the dependent test
results will also change. A dependent test can be detected byrunning tests in a default order once, and then rerunning thetests in reverse order or randomly [11]. If the test results aredifferent from those in the default order, then we can judge
the test to be a dependent test．In the present study, we detect
dependent tests by rerunning the test several times in random
order.
Workﬂow: Using the detection method described in theprevious section, we detect dynamic smells by the following
procedure:
1) Statically analyze the test code and collect method
information.
2) Embed coverage measurement instructions in the bytecode of test classes.
3) Run each instrumented test in default order.
4) If a test passes, then check whether the test is a rotten
green test. If a test fails, then check whether the test isa ﬂaky test.
5) After the tests are executed in default order, run the tests
in random order to detect dependent tests.
C. Implementation
Smell Detection: In order to detect dynamic smells, JTDog
ﬁrst generates ASTs of test codes using Eclipse JDT and

JTDog Plugin
sniff

User

1. Invoke

Gradle Java Plugin
testClasses

2. Invoke

3. Compile src/test
SourceSet

4. Retrieve SourceSet

JSON

Task

5. Detect dynamic smells- Rotten green test analysis (- Flaky test analysis ( )- Dependent test analysis ( )Test exec.

AST traversal

src files) test files
class files
classpath …

Coverage analysis

Fig. 1. Detection process using Gradle API

analyzes these ASTs to collect method information. JTDog
then instruments test classes with JaCoCo to measure thecoverages and run the tests in default order using JUnit.
Second, for each test method, if the test passes, then JTDog
checks whether it is a rotten green test. JTDog obtains the line
numbers of the assertions that the test method contains fromthe method information collected in advance. By analyzing thecoverage data to check whether the instruction in that line has
been executed, JTDog can determine whether the test contains
any unexecuted assertions. If the test includes method calls,
JTDog also checks the method recursively to ascertain whether
the test contains unexecuted assertions. In case of a test failure,
JTDog reruns the test multiple times, and if the test passes atleast once, then the test is considered to be a ﬂaky test.
After the tests are executed in default order, JTDog performs
a test execution in random order several times. Each test
execution is performed on a different JVM. For each test
method, the test result is compared with the result in a default
order, and if the results differ, then the test is determined to
be a dependent test. If the test has already been judged to bea ﬂaky test, then the test is treated as not being a dependent
test, because the test result will change independently of theorder.
Using Gradle API: In addition to production code andtest code, various other information, including class ﬁles andexternal library paths, is required in order to perform the dynamic smell detection process described above. Gradle, a build
tool, can automatically manage source folders and resolve
dependencies. By incorporating the dynamic smell detection
technique into Gradle, all of the information necessary forexecution can be retrieved from Gradle and high portability
can be achieved.
Figure 1 shows an overview of the detection process using
the Gradle API. In a Java project using Gradle, the Java plugin
is applied to compile and test. The sniff task provided by
JTDog depends on the testClasses task provided by the Javaplugin. This means that when the sniff task is executed, thetestClasses task is executed ﬁrst and performs preprocessing,such as ﬁle compilation.

The Java plugin also introduces SourceSets into the project.
A SourceSet is a logical group of Java source and resource
ﬁles and has information about the compilation of source
and resource ﬁles. By using a SourceSet, JTDog can retrieve
source ﬁles, class ﬁles, classpath, and so on. JTDog obtains
production codes and test codes from a SourceSet and collects
method information by generating and analyzing the ASTs oftest codes. The proposed plugin then acquires the test classﬁles and classpath from the SourceSet, and instruments andexecutes tests to obtain the test results and coverage data.
By combining the collected data, JTDog can detect dynamic
smells.
III. E VALUATION
In order to evaluate the portability of JTDog and its ability
to detect dynamic smells, we applied JTDog to Gradle projects
on GitHub.
A. Subjects
We selected projects on GitHub that met the following
criteria:• A project that uses the Gradle build tool, or a Maven
project that can be converted to a Gradle project without
problems using the conversion features of Gradle.
• The primary language is Java, and the Gradle Java pluginhas been applied.
• Contains at least one test.
• Uses JUnit as the testing framework.
• Has more than 100 star ratings.
Based on the above conditions, 150 projects were selected as
the target projects.
B. Experimental approach
For each project to be applied, we add a declaration forusing JTDog to its build.gradle ﬁle, as shown in Section 2.1.
Then, we run the sniff task and check the contents of the JSONﬁle in which the detection results are output.
C. Results
Portability: As a result of applying JTDog to 150 selected
projects, JTDog worked correctly in 122 projects, which isapproximately 81% of the total number of projects. This resultshows that JTDog can be applied to a variety of projects byonly adding one sentence to the build ﬁle to use this plugin.
For the 28 projects that failed to apply the JTDog plugin,
we visually checked for the cause of failure. The breakdown isshown in Table I. We investigated the reason for the failure due

TABLE IC AUSES OF APPLICATION FAILURES
Cause
Out of memory
Failed to create task
Incorrect detection result
Failed to read ﬁle# of projects
10
9
6
3

to an out of memory error, and found that this error was causedby the huge number of methods in the project. Since JTDog
collects and stores method information to detect dynamic
smells, the more methods there are, the more memory is used
to retain the information. In other words, the insufﬁcient heap
space is the cause of the out of memory error. By increasing
the heap space, we can prevent this problem. As for causes
other than out of memory errors, we do not fully understandthe reasons at present and need to analyze the causes of theseerrors in detail in the future. However, we consider that there
are few problems caused by the implementation of the tool,and that JTDog has high portability.
Detection Results: Table II shows the number of dynamic
smell detections in 122 projects in which JTDog worked properly and the number of projects that contains smells. Based on
the results, 55 out of 122 projects have dynamic smells, whichmeans that approximately 45% of the successfully applied
projects contain at least one dynamic smell. Moreover, therotten green test has the highest number of detections andmany projects include this smell. This is because the rotten
green test is a new test smell introduced in 2019 [3] andmost developers are not familiar with this smell. According
to Delplanque et al. [3], the rotten green test is worse than notest at all because the rotten green test gives developers thefalse impression that the test passed without any problems,
even though the test was poorly executed. Hence, tools thatcan detect the rotten green test are important.
We examine the tendency of each smell in each project.
Tables III, IV, and V show the number of each dynamic
smell detected in each project, for the top three projects. For
the rotten green test, the projects with the largest number ofdetections shown in Table III contained many smells with thesame cause. For instance, in the traccar project with the highest
number of detections, most of the 224 rotten green tests werecaused by not executing assertions in if statements. By using
JTDog, developers can learn which type of code is a rotten
green test, which can be useful for future development.
Regarding the number of ﬂaky test detections in Table
IV, the number of detections in the http-request project wasoutstanding, but was small in the other projects. This may be
TABLE IID ETECTION RESULTS
Dynamic smell
Rotten green test
Flaky test
Dependent test
Total

TABLE IIIROTTEN GREEN TESTDETECTION PROJECTS

Project
traccarelki
Apktool

#
224
110
58

# of detections
781
66
111
958

# of projects
48
5
13
55

TABLE IVF LAKY TESTTABLE VD EPENDENT TESTDETECTION PROJECTS

Project
http-request
java-jwt
spring-statem.

#
59
4
1

DETECTION PROJECTS

Project
ehcache3
http-request
micrometer

#
57
22
11

432
433
434
435
436
437
438
439
440
441
442
443

@Test
public void c1c3IsDisposed() {toV3Completable(Completable.complete()).subscribe(new CompletableObserver() {@Override
public void onSubscribe(Disposable d) {//this assertion is not executed.
assertFalse(d.isDisposed());
}
...
});
}

Fig. 2. Example of a detected rotten green test3

because, in addition to the fact that JTDog uses the simplest
detection method, RERUN, this method is supported by many
frameworks, such as Jenkins [14], and many projects deal withthis smell.
In all of the top three projects in Table V, the main cause of
the dependent test was sharing of static variables between tests.
Such tests may change the test results by changing the test
code, including adding tests that use the same static variables.
It is desirable to detect dependent tests at an early stage andtake countermeasures, so detection tools such as JTDog areimportant.
Illustrative Case: The test method c1c3IsDisposed() ofthe RxJavaInterop project shown in Figure 2 is a rotten green
test that JTDog detected. The onSubscribe() method (line437), an action of the event listener, has an internal assertionand is never executed. Therefore, the program is not veriﬁedby the assertion in line 439, and the test is insufﬁcient. Thiskind of callback function makes difﬁcult to conﬁrm whether
assertions will be executed or not. JTDog helps developers to
notice such problems that are easily overlooked.

Java programs. To detect dependent tests, Zhang et al. built
DTDetector [11], and Gambi et al. built PRADET [20]. Bellet al. proposed a new detection method for ﬂaky test andimplemented DeFlaker [12]. These are all standalone tools.
We integrated the detection functions of each dynamic smell
into a Gradle plugin. Integration into a build tool provides highportability and allows one tool to detect all dynamic smells,
which leads to increased productivity of developers.
There are integrations of test smell detection methods into
IDEs. Bleser et al. developed SoCRATES [21], an IntelliJ
plugin that can detect six types of test smells. Lambiase etal. created an IntelliJ plugin, DARTS [22], which can detect
three types of test smells and perform automatic refactoring.
Eclipse plugins have also been created. Baker et al. built TReX[23], and Santana et al. built RAIDE [24]. These plugins
provide support for refactoring. Koochakzadeh and Garousi
developed TeReDetect [25] and TeCReVis [26]. These tools
are integrated into an Eclipse plugin called CodeCover [27].
By integrating test smell detection techniques into the IDE,
they reduce the cost of test smell detection for users. Similarly,
JTDog helps users to detect test smells easily by integrating
dynamic test smell detection methods into Gradle.
V. C ONCLUSIONSIn this paper, we introduced JTDog, a Gradle plugin for
detecting dynamic test smells. In order to evaluate the portability of JTDog and its ability to detect dynamic smells, weapplied JTDog to 150 projects on GitHub. JTDog detected
958 dynamic smells from 55 projects. In the future, we intend
to improve the dynamic smell detection method and to extend
JTDog to detect both static and dynamic smells. Moreover, weplan to enable JTDog to analyze tests from other Java testing
frameworks, such as TestNG.

IV. R ELATED W ORK
Many detection tools have been proposed for test smells.
Breugelmans and Rompaey developed TestQ [15], a tool toquantify test smells. Greiler et al. introduced new test smells
associated with test ﬁxtures and created a detection tool,
TestHound [8]. In a subsequent study, the authors extended
TestHound to analyze Git and SVN repositories and released
TestEvoHound [16]. Palomba et al. built TASTE [17], a tool
to detect three types of test smells by textual analysis. Bavota
et al. [18] created an unnamed test smells detection tool that
detects nine types of test smells, and Peruma et al. proposed
tsDetect [9], which can detect 19 types of test smells. All ofthese tools are capable of detecting only static smells and arecomplementary to JTDog.
Detection tools have been created for each of the three test
smells that we have classiﬁed as dynamic smells. Delplanqueet al. proposed a rotten green test as a new test smell
and developed DrTest as a detection tool [3]. Since DrTest
targets programs written in the Pharo language, Martinez etal. developed RTj [19], a rotten green test detection tool for
3 https://github.com/akarnokd/RxJavaInterop/blob/3.x/src/test/java/hu/
akarnokd/rxjava3/interop/RxJavaInteropTest.java#L432

ACKNOWLEDGMENTS
This research was partially supported by JSPS KAKENHI
Japan (Grant Number: JP21H04877)
R EFERENCES[1] A. Deursen, L. Moonen, A. Bergh, and G. Kok, “Refactoring Test Code,”in International Conference on Extreme Programming and Flexible
Processes in Software Engineering, 2001, pp. 92–95.
[2] G. Meszaros, xUnit test patterns: Refactoring test code.
Pearson
Education, 2007.
[3] J. Delplanque, S. Ducasse, G. Polito, A. P. Black, and A. Etien, “Rotten
Green Tests,” in International Conference on Software Engineering,
2019, pp. 500–511.
[4] B. V. Rompaey, B. D. Bois, and S. Demeyer, “Characterizing theRelative Signiﬁcance of a Test Smell,” in International Conference on
Software Maintenance, 2006, pp. 391–400.
[5] G. Bavota, A. Qusef, R. Oliveto, A. Lucia, and D. Binkley, “Anempirical analysis of the distribution of unit test smells and their impact
on software maintenance,” in International Conference on Software
Maintenance, 2012, pp. 56–65.
[6] G. Bavota, A. Qusef, R. Oliveto, A. Lucia, and D. Binkley, “Are
Test Smells Really Harmful? An Empirical Study,” Empirical Software
Engineering, vol. 20, no. 4, pp. 1052–1094, 2015.
[7] A. Peruma, “What the Smell? An Empirical Investigation on theDistribution and Severity of Test Smells in Open Source Android
Applications,” Master’s thesis, Rochester Institute of Technology, 2018.

[8] M. Greiler, A. van Deursen, and M. Storey, “Automated Detection of
Test Fixture Strategies and Smells,” in International Conference on
Software Testing, Veriﬁcation and Validation, 2013, pp. 322–331.
[9] A. Peruma, K. Almalki, C. Newman, M. Mkaouer, A. Ouni, and
F. Palomba, tsDetect: An Open Source Test Smells Detection Tool.
Association for Computing Machinery, 2020, pp. 1650–1654.
[10] Q. Luo, F. Hariri, L. Eloussi, and D. Marinov, “An Empirical Analysis
of Flaky Tests,” in International Symposium on Foundations of Software
Engineering, 2014, pp. 643–653.
[11] S. Zhang, D. Jalali, J. Wuttke, K. Muşlu, W. Lam, M. Ernst, and
D. Notkin, “Empirically Revisiting the Test Independence Assumption,”in International Symposium on Software Testing and Analysis, 2014, pp.
385–396.
[12] J. Bell, O. Legunsen, M. Hilton, L. Eloussi, T. Yung, and D. Marinov, “DeFlaker: Automatically Detecting Flaky Tests,” in International
Conference on Software Engineering, 2018, pp. 433–444.
[13] K. Herzig and N. Nagappan, “Empirically Detecting False Test Alarms
Using Association Rules,” in International Conference on Software
Engineering, 2015, pp. 39–48.
[14] Jenkins RandomFail annotation, “@RandomlyFails,” https://github.com/
jenkinsci/jenkins-test-harness/blob/master/src/main/java/org/jvnet/
hudson/test/RandomlyFails.java, 2017.
[15] M. Breugelmans and B. V. Rompaey, TestQ: Exploring Structural andMaintenance Characteristics of Unit Test Suites, International Workshop
on Advanced Software Development Tools and Techniques, 2008.
[16] M. Greiler, A. Zaidman, A. van Deursen, and M. Storey, “Strategies
for avoiding text ﬁxture smells during software evolution,” in Working
Conference on Mining Software Repositories, 2013, pp. 387–396.
[17] F. Palomba, A. Zaidman, and A. D. Lucia, “Automatic Test Smell
Detection Using Information Retrieval Techniques,” in International
Conference on Software Maintenance and Evolution, 2018, pp. 311–
322.
[18] G. Bavota, A. Qusef, R. Oliveto, A. D. Lucia, and D. Binkley, “Anempirical analysis of the distribution of unit test smells and their impact
on software maintenance,” in International Conference on Software
Maintenance, 2012, pp. 56–65.
[19] M. Martinez, A. Etien, S. Ducasse, and C. Fuhrman, “RTj: A Java
Framework for Detecting and Refactoring Rotten Green Test Cases,”in International Conference on Software Engineering, 2020, pp. 69–72.
[20] A. Gambi, J. Bell, and A. Zeller, “Practical Test Dependency Detection,” in International Conference on Software Testing, Veriﬁcation andValidation, 2018, pp. 1–11.
[21] J. D. Bleser, D. D. Nucci, and C. D. Roover, “SoCRATES: Scala Radarfor Test Smells,” in Symposium on Scala, 2019, pp. 22–26.
[22] S. Lambiase, A. Cupito, F. Pecorelli, A. D. Lucia, and F. Palomba, “JustIn-Time Test Smell Detection and Refactoring: The DARTS Project,” inInternational Conference on Program Comprehension, 2020, pp. 441–
445.
[23] P. Baker, D. Evans, J. Grabowski, H. Neukirchen, and B. Zeiss, “TRex- The Refactoring and Metrics Tool for TTCN-3 Test Speciﬁcations,”in Testing: Academic & Industrial Conference - Practice And ResearchTechniques, 2006, pp. 90–94.
[24] R. Santana, L. Martins, L. Rocha, T. Virgı́nio, A. Cruz, H. Costa, andI. Machado, “RAIDE: A Tool for Assertion Roulette and Duplicate
Assert Identiﬁcation and Refactoring,” in Proceedings of the 34th
Brazilian Symposium on Software Engineering, 2020, pp. 374–379.
[25] N. Koochakzadeh and V. Garousi, “A Tester-Assisted Methodology for
Test Redundancy Detection,” Advances in Software Engineering, vol.
2010, 2010.
[26] N. Koochakzadeh and V. Garousi, “TeCReVis: A Tool for Test Coverage
and Test Redundancy Visualization,” in Testing – Practice and ResearchTechniques, 2010, pp. 129–136.
[27] CodeCover, http://codecover.org/index.html.