SUGISAKAMASANORI

Journal of Advances in Artificial Life Robotics
Vol. 4(1); June (2023), pp. 35–40
ON LINE ISSN 2435-8061; ISSN-L 2435-8061
https://alife-robotics.org/jallr.html

Research Article

Prototype of the Framework CATdd to Support Continuous
Development in Test Driven Development
Takeaki Miyashita1, Tetsuro Katayama1, Yoshihiro Kita2, Hisaaki Yamaba1, Kentaro Aburada1, Naonobu Okazaki1

1
Department of Computer Science and Systems Engineering, Faculty of Engineering, University of Miyazaki, Gakuen-kibanadai nishi, Miyazaki, 889-2192,
Japan
2
Department of Information Security, Faculty of Information Systems, Siebold Campus, University of Nagasaki, 1-1-1 Manabino, Nagayo-cho, Nishi-Sonogigun, Nagasaki, 851-2195, Japan

ARTICLE INFO
Article History
Received 25 Nobember 2022
Accepted 04 September 2023
Keywords
Source code generation
Test driven development (TDD)
Large language model (LLM)

ABSTRACTTDD is a development methodology to improve software quality. In this study, we prototype theframework CATdd (Continuous Automated Test Driven Development), which supports
continuous development with TDD. CATdd reduces the time of implementation step in TDD bygenerating new source code that passes tests using LLM. CATdd maintains consistency with othersource code in the project by using existing source code and related source code to prompt. As aresult of the evaluation experiments, the time for the implementation step in TDD was reducedby 94% for a simple task and by 56% for another more complex task. In conclusion, CATdd isuseful to support continuous development.

© 2022 The Author. Published by Sugisaka Masanori at ALife Robotics Corporation Ltd.
This is an open access article distributed under the CC BY-NC 4.0 license(http://creativecommons.org/licenses/by-nc/4.0/).

Such inconsistencies lead to increased development timeand bugs in continuous development. Hence, to support
continuous development, it is necessary to resolve suchinconsistencies and maintain consistency in the project.
In this study, we prototype the framework CATdd(Continuous Automated Test Driven Development),
which supports continuous development with TDD.
CATdd reduces the time of implementation step in TDD
by generating new source code that passes tests using
LLM. CATdd maintains consistency with other sourcecode in the project by using existing source code andrelated source code to prompt. In this prototype, we use
python to implement CATdd. CATdd only supports a
project that is implemented in C++ and runs tests in
GoogleTest.

1. Introduction
Software quality and development efficiency havebecome more important [1]. Test Driven Development(TDD) is a development methodology to improve
software quality [2]. In TDD, developers repeat a seriesof test design step, implementation step, and refactoring
step to make test cases and source code better. By
reducing the time of the implementation step in TDD,
developers can focus more on test [3].
To improve the efficiency of software development,
source code generation using the Large Language Model(LLM) has been studied [4], [5]. However, source code
generated by LLM may have inconsistencies with othersource code in the project. Examples of inconsistencies
are duplication of processing between source codes,
mismatch of arguments in function calls, and mismatchof coding styles. In addition, if the LLM ignores
refactoring by the developer and overwrites the source
code, the developer will have to repeat the refactoring.

Corresponding author’s E-mail: kat@cs.miyazaki-u.ac.jp, miyashita@earth.cs.miyazaki-u.ac.jp, kita@sun.ac.jp, yamaba@cs.miyazaki-u.ac.jp,
aburada@cs.miyazaki-u.ac.jp, oka@cs.miyazaki-u.ac.jp

35

Fig. 1 The Structure of CATdd
2. CATdd

3.2. Test Result Analyzer

CATdd is a framework to support continuous softwaredevelopment with TDD. CATdd runs tests on the target
project and generates source code that can pass the tests
based on the source code that failed the tests, maintaining
the consistency in the project by using LLM. In thisprocess, CATdd refers to another source code that is
loaded by the source code to be tested as a related sourcecode. To generate source code with CATdd, thedeveloper must pre-set 5 data in catdd.yaml: the sourcecode path, the test execution path, the test executioncommand, the programming language used in the project,
and the language of comments when generating source
code.

Test Result Analyzer extracts data about failed test cases
from the test results received from the Test Runner. Test
Result Analyzer extracts failed test cases that match thepre-defined regular expression patterns for assertion
errors and the 3 errors: “No such file”, “has no member”,
and “undefined”. The extracted test cases data is sent to
Source Code Searcher and Test Code Analyzer.
3.3. Source Code Searcher
Source Code Searcher searches for source files based on
the class names that failed to pass the test from theinformation received from the Test Result Analyzer. If asource file is found, the header file that the source fileincludes is also searched. The source code of the
discovered file is passed to the Source Code Generator.
In this prototype, only the header file of the class to betested was the target of the search. If no files are found,empty character is sent to the Source Code Generator.

3. Implementation
The structure of CATdd is shown in Fig. 1. As shown in
Fig. 1, CATdd has 5 processing sections: Test Runner,
Test Result Analyzer, Source Code Searcher, Test CodeAnalyzer, and Source Code Generator. The behavior ofeach processing section is described below.
3.1. Test Runner
Test Runner runs tests on the project supported by
CATdd using the test execution path and test executioncommands in catdd.yaml, and redirects the test results toa log file. The log file of the test results is sent to the Test
Result Analyzer.

36

Implement the source code for class {Class Name of Failed Test}

TEST(calculateMileageTest, calculateMilage)that satisfies the following conditions using {Programing{

Language} with {Language of Comments} comments.

double radius = 50;- Only the source file must be implemented out of the two files,

double rightAngle = 10;

header file and source file.

double leftAngle = 20;- Satisfy all of the following test cases. However, it is notdouble rightWheel = 2 * rightAngle * radius * M PI / 360;

necessary to run the tests.

double leftWheel = 2 * leftAngle * radius * M PI / 360;

{Test Code for Failed Test}

double expected = (rightWheel + leftWheel) / 2;

- Based on the following source code.

double actual = Mileage::calculateMileage(rightAngle,{Existing Source Code}leftAngle);- Based on the following header file.

EXPECT DOUBLE EQ(expected, actual);{Related Source Code}}Fig. 2. The Syntax of Prompt

Fig. 3. Part of the Test Codecases, relevant source code, source code that failed topass existing tests. In the reducing data for failed test
cases, at least one of the failed test cases is included in
the prompt. When the number of tokens is below thelimit, Source Code Generator generates the prompt andsends it to the OpenAI API for source code generation.
Finally, Source Code Generator writes the generated
source code to a file on the project. If the number oftokens is over the limit after reducing all the data, Source
Code Generator notifies the developer and exits theprocess.

3.4. Test Code Analyzer
Test Code Analyzer extracts the test code for the failed
test cases based on the test code line numbers received
from the Test Result Analyzer and sends it to the Source
Code Generator.
3.5. Source Code Generator
Source Code Generator generates prompts for sourcecode that pass testing and maintain consistency with therelated source code, based on code received from Source
Code Searcher and Test Code Analyzer. Then, Source
Code Generator sends the generated prompts to the
OpenAI API [6] for generating the source code. This
prototype uses the text-davinci-003 model. Source Code
Generator generates 2 files, a source file (.cpp) and aheader file (.h), for the target of the failed test.
The syntax of the prompt is shown in Fig. 2. The data
composing the prompt, the programming language andthe language of comments, are taken from theconfiguration file catdd.yaml, the class name of the failed
test target, the existing source code, and the related
source code are taken from the Source Code Searcher,and the test code of the failed test is taken from the Test
Code Analyzer.
The sending the prompt to the OpenAI API has aproblem, the request is rejected if the number of tokens
is over the limit [6]. To solve the problem, Source Code
Generator ranks the data that makes the prompt in orderof priority and reduces them in the order of decreasingpriority. The data is reduced in the following order:
comments in the relevant source code, comments in thesource code that failed to pass existing tests, failed test

4.

Application Example

To confirm that the prototype CATdd works as expected,using an application example. In an empty project
supported by CATdd, test code and header files weremanually described for the Mileage class that calculates
the movement distance from the rotation angles of bothwheels, and generated source code by CATdd. A part of
the test code and the header file used to generate thesource code are shown in Fig. 3 and Fig. 4, and thesource file generated by CATdd is shown in Fig. 5. After
source code generation, the project passed all tests. Thisconfirms that CATdd can generate source code that canpass tests. Also, Fig. 5 shows that CATdd implemented
the private function “calculateWheelMileage()”, which is
not described in the test code, by using the header file
shown in Fig. 4 for the source code generation. Thisconfirms that CATdd can maintain consistency withother source code in the project by using source code
related to the target of generation as input.

37

#ifndef MILEAGE_H

Table 1. The Average Implementation Time per Person#define MILEAGE_H

Method#include <cmath>class Mileage {ManualCATdd

public:/*** @brief Calculate mileage of both wheel

1st Task
89m33s
5m19ss

2nd Task
101m16s
44m21s

The first task is to implement a Mileage class that
calculates the movement distance from the rotation
angles of both wheels. This task is simple and can be
implemented with about 30 lines of source code for 5 test
cases. The second task is to implement the member
function calculatePid() of the Pid class that executes PID
control. This task is more complex and can be
implemented with about 20 lines of additional source
code, for 8 test cases and about 50 lines of existing sourcecode. The test code was created by the experimenter andall test cases were commented out.
In the experiment, subjects executed the following
steps: uncomment a test case, run the test to check theresults, implement source code that pass the test, refactor
if necessary, and repeat these steps until the task iscompleted. Additionally, after completing both tasks,
they submitted feedback on using CATdd.

* @param rightAngle Right motor angle [deg]* @param leftAngle Left motor angle [deg]* @return Mileage of both wheel [mm]*/static double calculateMileage(int rightAngle, int leftAngle);
private:static const double RADIUS;
Mileage();/*** @brief Calculate mileage of a wheel* @param angle Motor angle of a wheel [deg]* @return Mileage of a wheel [mm]*/static double calculateWheelMileage(int angle);};#endif5.1. Evaluation of Implementation Time

Fig. 4. Described Header File

The average implementation time per person for eachcombination of task and implementation method isshown in Table 1. Table 1 shows that CATdd reduced thetime of implementation by 84m14s (94%) for the firsttask and by 56m55s (56%) for the second task. This
indicates that CATdd is useful for reducing time of
implementation step in TDD.

// Mileage.cpp// Implementation of Mileage class#include "Mileage.h"// Constructor
Mileage::Mileage() {}double Mileage::calculateMileage(int rightAngle, int leftAngle) {double rightWheel = calculateWheelMileage(rightAngle);
double leftWheel = calculateWheelMileage(leftAngle);
return(rightWheel + leftWheel) / 2;

5.2. Evaluation of the Generated Source Code}During the experiment, CATdd generated source codethat passed most tests. On the other hand, CATdd
generated source code that failed to consider the patternand did not pass the test, even though there was a test casefor division by zero. Also, during the experiment, thegenerated source code was sometimes not source code orcontained errors that CATdd did not support. Generating
incorrect source code increases developer effort anddevelopment time. To improve the quality of generated
source code, it is necessary to increase the quantity andquality of information used for source code generation.

double Mileage::calculateWheelMileage(int angle) {return 2 * angle * RADIUS * M_PI / 360;}

Fig. 5. Generated Source Code By CATd
5. Evaluation
We evaluate the effective of the time of implementation
step in TDD by comparing the implementation with andwithout CATdd in the experiment. 7 students majoring incomputer science and engineering, 4 graduate students
and 3 undergraduates, participated in the experiment as
subjects. They solved 2 tasks using TDD. 4 subjects used
CATdd for only the first task, and other subjects used
CATdd for only the second task.

5.3. Evaluation of Usefulness by Questionnaire
Subjects answered what was good and bad about CATdd
after the task was completed.

38

As good points, 5 of the 7 subjects mentioned that thetask was easier, and 3 mentioned that they were able toimplement it faster. In addition, the comments generated
automatically were useful, and the number of
implementation errors was reduced. On the other hand,as bad points, 3 of the 7 subjects were able to implement
without understanding the source code generated
automatically, and 2 subjects found it difficult tounderstand what changes had been made due to theautomatic generation. In addition, they also mentioned
the time and effort required to delete unnecessary code,change variable names, etc., which is not required inmanual development.
From these answers, it can be said that CATdd isuseful in reducing development time in TDD.

6. Conclusion
We have prototyped the framework CATdd, which
supports continuous development with TDD. Using theapplication example, we have confirmed that CATdd canmaintain consistency with other source code in the
project. As a result of the evaluation experiments, thetime for the implementation step in TDD was reduced by
94% for simple task and by 56% for more complex task.
In addition, the questionnaire showed that CATdd isuseful in reducing implementation time and human error.
In conclusion, CATdd is useful to support continuous
development in TDD.
Our future tasks are shown below.
 Improve quality of generated source code Extend range of related source code support
 Guide to automatically generated code

5.4. Related Research

References

One research that automatically generates source code is
CodeT developed by Chen et al [4]. CodeT generates
source code using natural language requests as input. In
addition, it also generates test code and evaluates thegenerated source code when it generates source code.
However, CodeT needs to re-generate source code withnew requirements as input when the requirements arechanged. On the other hand, CATdd uses the existingsource code as input and can generate new source code
maintaining the changes made by the developer's
refactoring. Since minor modifications by the developer
are retained in the next source code generation, CATdd
can support continuous development.
One research of APR (Automated Program Repair),which automatically corrects test failures and errors, isthe work of Chunqiu et al [5]. In this study, they proposed
a conversational APR that alternates between generating
patches and executing tests using LLMs. Conversational
APR generates new patches by taking as input a program
containing bugs, test cases, and previously generated
patches and test results for those patches. Repeated input
of previously generated patches and test results to themodel prevents the generation of patches that havealready been generated. However, since only one
program is used as input, it does not support processes
that can be realized by multiple programs. On the other
hand, CATdd uses the related source code loaded in thesource code as input for source code generation, in
addition to the failed tests and the source code to betested. By using the related source code, CATdd
improves the possibility of generating source code withconsistency within the project.

1. “BBC: What went wrong inside Boeing’s cockpit?”
https://www.bbc.co.uk/news/extra/IFtb42kkNv/boeingtwo-deadly-crashes#group-System-failurezAFs52mVNd(Accessed 2023-9-3).
2. Rafique, Y., and Miˇsi´c, V., “The Effects of TestDrivenDevelopment on External Quality and Productivity:
AMeta-Analysis”, IEEE Transactions on Software
Engineering, Vol.39, No.6, pp.835-856, 2013.
3. Anwer, F., Aftab, S., Waheed, U., et al., “Agile
SoftwareDevelopment Models TDD, FDD, DSDM, andCrystalMethods: A Survey”, International Journal ofMultidisciplinarySciences and Engineering, Vol.8, No.2
pp.1-10, 2017.
4. Chen, B., Zhang, F., Nguyen, A., et al., “CodeT: Code
Generation with Generated Tests”, The EleventhInternational Conference on Learning Representations
2023 poster, 19p. 2023.
5. Xia, C.S. and Zhang, L., “Conversational AutomatedProgram Repair”, arXiv:2301.13246, 11p, 2023
6. “OpenAI:
OpenAIAPI
Documentation”,
https://platform.openai.com/docs/introduction, (Accessed2023-7-31)

Authors Introduction
Mr. Takeaki Miyashita
He received the Bachelor's degree inengineering (computer science andsystems engineering) from the
University of Miyazaki, Japan in 2022.
He is currently a Master's student in
Graduate School of Engineering at the
University of Miyazaki, Japan. His
research interests software testing andsoftware quality.

39Dr. Tetsuro Katayama

Dr. Naonobu Okazaki
He received his B.S, M.S., and Ph.D.
degreesinelectricalandcommunication engineering fromTohoku University, Japan, in 1986,
1988 and 1992, respectively. He
joined the Information Technology
Research and Development Center,
Mitsubishi Electric Corporation in1991. He is currently a Professor with
the Faculty of Engineering, University of Miyazaki since
2002. His research interests include mobile network andnetwork security. He is a member of IPSJ, IEICE andIEEE.

He received a Ph.D. degree inengineering from Kyushu University,
Fukuoka, Japan, in 1996. From 1996 to
2000, he has been a Research
Associate at the Graduate School ofInformation Science, Nara Institute of
Science and Technology, Japan. Since
2000 he has been an Associate
Professor at the Faculty of Engineering, Miyazaki
University, Japan. He is currently a Professor with theFaculty of Engineering, University of Miyazaki, Japan.
His research interests include software testing and quality.
He is a member of the IPSJ, IEICE, and JSSST.
Dr. Yoshihiro Kita

He received a Ph.D. degree in systems
engineering from the University ofMiyazaki, Japan, in 2011. He iscurrently an Associate Professor with
the Faculty of Information Systems,
University of Nagasaki, Japan. His
research interests include software
testing and biometrics authentication.

Dr. Hisaaki Yamaba
He received the B.S. and M.S. degrees
in chemical engineering from theTokyo Institute of Technology, Japan,
in 1988 and 1990, respectively, and the
Ph D. degree in systems engineeringfrom the University of Miyazaki, Japan
in 2011. He is currently an AssistantProfessor with the Faculty ofEngineering, University of Miyazaki, Japan. His researchinterests include network security and user authentication.
He is a member of SICE and SCEJ.
Dr. Kentaro Aburada

He received the B.S., M.S, and Ph.D.
degrees in computer science andsystem engineering from the
University of Miyazaki, Japan, in
2003, 2005, and 2009, respectively.
He is currently an Associate
Professor with the Faculty ofEngineering,
UniversityofMiyazaki, Japan. His research interests include computer
networks and security. He is a member of IPSJ and IEICE.

40