DEIM Forum 2017 P7-3

Evaluating Programming Ability by Using a Visual Contents ComparisonMethodDick MARTINEZ CALDERON†

Yukinobu MIYAMOTO‡

Hidenari KIYOMITSU†

Kazuhiro OHTSUKI††Graduate School of Intercultural Studies, Kobe University 1-2-1 Rokkodaicho, Nada-ku, Kobe, 657-8501 Japan‡Graduate School of Information Technology, Kobe Institute of Computing

2-2-7 Kanocho, Chuo-ku, Kobe, 650-0001 Japan
E-mail: †dick.martinez@gmail.com,†{ohtsuki, kiyomitu}@kobe-u.ac.jp‡miyamoto@kic.ac.jp

Abstract Until now, Paper tests and practical programming exercises have been widely used to evaluate programming
abilities but in recent years, professionals in different fields have become able to do programming by using simplified softwaretools; as a consequence of this, they have become able to understand and do programming in a general or panoramic way. This
paper proposes a new method for evaluating programming abilities based on the comparison of visual output (pictures,animations) produced by 2 or more programming samples. By comparing these output contents a student must decide which oneof the programs producing them is more difficult to build with programming than the other, or, if the difficulty is similar for bothof them.
Keyword Computer Science Education ， Programming Training ， Student Assessment ， Graphic Design ， Software
Engineering

1. Panoramic Understanding of Programming

paid to the evaluation of programming ability in this wide

Software development has changed drastically during

range of knowledge areas, and particularly, the evaluation

the last two decades; more and more people not involvedof the different ways professionals not related within professional software development have become able toSoftware Development could be able to understand anddo programming and new resources to make programming

apply programming skills according to their knowledge; weeasier have been created. For example: code samples andcall this a Panoramic Understanding of Programming.

tutorials are being uploaded to the web and used throughThis new way to understand programming is different

copy-pasting; a large number of algorithms are constantly

than the programming perception applied by sof tware

being converted into libraries and made widely available,

developers or programmers. To be more specific, this newso to find the best-suited function within libraries has

understanding is not related to programming language

become an important task; and several visual software

grammar, code writing, reading or debugging, or practicaldevelopment tools and languages, where the programming

performance at making programs, but instead, it ’s morecode is hidden and it can be applied with “just a click” arerelated to an awareness of how to build a program or how

being developed.

to use structures or patterns in a specific programming

Additionally, the background and learning modes ofsituation without knowing in depth the foundation of thosepeople using programming in their fields or ar eas ofstructures, or how to produce them using a specificknowledge is becoming more diverse. Nowadays Software

language or to do them from scratch.

developers and IT students learn to use code samples,

2. A Method to Identify New Abilities onlibraries and interfaces as a complement to traditional ways

Programming.

to do programming, while students from other fields not

The objective of our research is to identify and measure

related with software development such as arts or businessabilities related with the aforementioned “Panoramicare learning programming through authoring tools andUnderstanding of Programming” in students from different

simplified programming languages.

fields. To perform this, we propose a Programmed Visual

Even when many studies have been proposing newContents Comparison (PVCC) method based on thesystems and software tools oriented to reduce the gap

comparison of 2 or more output pictures (including

between those different fields wh en learning skills on

animations, interactive images, graphs, text) produced byprogramming, up to now, far too little attention has been

programming samples, if this comparison is showed to atested person, he is requested to decide which on e of thepictures is more difficult to build with programming thanthe other, or, if the difficulty is similar for both of them.
The correct answer for a question is defined by the mostdifficult programming process (algorithm) in both samples;basically, the student needs to identify this process toprovide the right answer to each question.
The person answering to any of the proposed samples
comparisons is asked to think about each comparison using
anyexperienceandknowledge

hecould

have

Figure 2

Comparison including Hidden Line Removalonprocessprogramming, as little as it could be, regardless of the tools

Those students of programming knowing how difficult itor programming languages he could know. The following

is to draw circles the way they are displayed on the sample

examples will allow us to explain more in detail thismarked with (1) without using any libraries, or by using

aspect:older programming languages (closer to machine language),would surely understand the difficulty of the Hidden Line
Removal process used on the sample marked with (1).
By contrast, those students who are used to program withsimplified programming languages, or by using libraries,
would probably answer that the difficulty is similar sincewith those languages both samples can be produced byusing the same code changing only its parameters. Thesestudents are surely unaware of what kind of algorithm isthe Hidden Line Removal and how it is applied.
Based on this method We built a Web Testing, whereFigure 1

Comparison Including Iteration process

Questions including three types of samples: Static Pictures,
Animated Graphics and Controlled by Mouse (sampleFigure 1 shows a comparison where both samples arebuilt by using the same code (an iteration process)objects can be moved or changed by hovering and clicking)were arranged.

changing only its parameters. In this sense, the correct
answer for this question was established to be: thedifficulty is similar.
We would expect students who understand how theiteration process is applied on both samples to answer thedifficulty is similar, since they would surely identify thatboth samples are built by using the same program onlychanging its parameters.
In the other hand, those students choosing one sample
over the other as their answer are probably unaware of thespecific programming process used to build both samples(iteration) and would probably consider their difficulty
based more on screen presentation issues (e.g. scale,distance between objects, visual impress ion) than on how
they are programmed.
Fig. #2 shows a Question where the sample marked with
(1) uses a Hidden Line Removal process to draw circles,
while the program of the sample marked with (2) doesn ’t
use this process, therefore the correct answer for t hisquestion was decided to be: the sample marked with (1).

Figure 3

Example of a Question as displayed on the Web
Testing System

Table 1.
Do you think this test was
useful for you to know yourow n ability inprogrammi ng?

Feedback from the Verification Test.

Do you think that by reading theex planation about the right answ ers
and thinki ng the answ ers of thistest, you have learned or review edanything about progra mmi ng?

Did you find this test moreenjoyable w hen co mparedwith usual paper
programmi ng skills tests?

70%

Ye s , I t h i n k s o74%

Ye s , I t h i n k s o89%

Ye s , I f o u n d i tmore enjoyable10%

I don’t think it was
useful12%

I don’t think I have learnedor reviewed anything aboutprogramming at all5%

I didn’t enjo y it atall

20%

I don’t know

14%

I don’t know

6%

I don’t know

Figure 3 shows an example of how a Question was

In this respect, our proposed method has proven to havedisplayed on screen; in this case both samples werepotential to identify the aforementioned kind of abilities.

Animated Graphics, the sample on the left draws and erases

We applied this method in a test performed with morea circle each frame, while the sample on the right draws

than 200 students of different fields (reportedly: Gamecircles each frame without erasing them.

Design,3. Potential of the Proposed Method to Evaluate

development), all of them provided feedback regarding

GraphicDesignandIT including

softwarethree items which results are described in the Table 1 andProgramming Abilities.
As it was mentioned before, through using this method
we want to identify abilities related to what we defined aswill be discussed in the following paragraphs:
70% of the students indicated that they were able to“panoramic” understanding of programming, or in othermeasurewords, a general awareness of programming through whichthemselves with this method.

a person can effectively make programs by putting together

theirabilityonprogramming

bytestingThis result suggests that, in spite of their field ordeepknowledge level, the students who answered this test wereknowledge about programming (as software developers

able to figure out how each programming sample of ahave) or having learned pro gramming by other ways

comparison could be done and based on their answer theydifferent than those applied in software development

knew if they were able to handle or not a particular aspectcourses.

of programming of those evaluated (quote FIE article)

severalexternal

resources,

without

havingaThe proposed method, then, has the potential to beHowever, their answer to the comparisons of the test

applied in order to measure abilities related with thiseven when “correct” couldn’t have been appropriate fromgeneral awareness of programming; for example:

a common programming perspective, or in the opposite side,The ability to know how (and where, whateven when “wrong” could have been given according to alibrary or snippet) to obtain a specific “part” orcommon programming point of view.

“piece” of code to make something work,For example: Even though some students answering thedepending or not of the language.

problem of Figure 1 probably didn’t have an idea of what

The ability to “connect” or “replace” code parts“nestedfollowing the logic of an already written(programming) thinking different to “Nested iteration” toprogram (perhaps written by another person).

figure out that both samples have a somewhat similar

The ability to grasp fundamental programming

difficulty if done with programming.

iteration”

was,

they

wereabletouseastructures (e.g. loops, conditional structures

On the other hand, if fig 1 example is viewed from a tooetc.) intuitively by understanding more complex

strict programming point of view, the second programming

structures (functions, objects).

sample could be a little more difficult than the first sample.

The ability to understand deep concepts ofSome students probably thought on the actual difficulty

programming (e.g. resourciveness) from output

required in programming to draw squares on screen andelements like graphics or animations, evenadjust them in a grid with proportional interspaces; andwhen not being able to do a recursive function.

since the first sample has less squares it can take less timeon doing it, and also require less computing resour ces (lessevaluation standards or scales for each measured abilityprocessing, less memory).

related with Panoramic Understanding of Programming,

This way of thinking, even when belonging probably toand enhance question’s classification, probably proposing

a person who has a deep knowledge of programming, could

different types of tests reach ing different level of abilities

be considered as a “panoramic understanding” because thefor the same school year.

student is also considering a particular way to think about

A natural progression of this work is to perform more

the program depending on the knowledge he could have.

tests using each time more complete and precise questions

In addition to the percentage of students that considered

and keep verifying their effectiveness. Future trials of thethe test useful to know their own programming ability, atest based on the proposed Programmed Visual Contents

74% of students considered that they learned a lot by

Comparison Method should assess effectively the desired

knowing what the right answer was for each one of theprogramming abilities.

comparisons. And interestingly a 53% of that total ofReferences

students belongs to fields learning classic programming[1] Kursat Ozenc, F., Miso, K., Zimmerman, J., Oney, S.
and Myers, B.: “How to Support Designers in Getting
Hold of the Immaterial Material of Software”,
Proceedings of the 2010 SIGCHI Conference on
Human Factors in Computing Systems (CHI '10) , pp.
2513-2522 (2010)[2] Ko, A., Myers, B. and Aung, H.H.: “ Six Learning
Barriers in End-User Programming Systems ”,
Proceedings of the 2004 IEEE Symposium on Visual
Languages - Human Centric Computing (VLHCC '04) ,pp. 199-206 (2004)[3] Martinez Calderon, D., Kin, M., Kiyomitsu, H.,
Ohtsuki, K. and Miyamoto, Y.: “An Evaluation
MethodforPanoramic
UnderstandingofProgramming by Comparison with Visual Examples ”,
Proceedings of the 2015 Frontiers in Education
Conference (FIE 2015), pp.511-518 (2015)

theory and how to do algorithms (namely: Game design, ITand software development).
This result could indicate that th ese students who know
programming actually figured out that there are other
different ways to do (or to think) the programs they arealreadyusedtoanswer,

challenging

theirown

preconception of how a program can be built, and changing
the way to look for the difference between two programs.
In this sense, if a student of programming is capable ofgetting out of preconceived knowledge patterns andconsideroptimize

consideredasan

his solution,ability

thisrelatedtocould

also

be

a

panoramic

understanding of programming.
By comparing the test based on the proposed methodwith a usual programing proficiency test, namely, written
tests or practical “hands-on” tests, an 88% of the evaluatedstudents found it to be enjoyable.
This result suggests that, the whole experience ofcomparing two programming samples, besides giving thestudent the possibility to challenge their way to think abouta problem (in this case a programming problem), is varied
enough to provide new contents and new challenges every
time and, since the comparisons are made to be answeredin a fast pace (approximately 2 -3 minutes per comparison),and there is no need to remember language syntax orstructures.

4. Future Work.
Further studies need to be carried out in order toestablish if the proposed Programmed Visual Contents
Comparison Method can effectively measure programming
ability. A greater focus on establishing how to measure
students specific programming abilities could produce
interesting findings that account more to validate thismethod.
Future improvement should also focus on building