# A Survey of the Prior Programming Experience of Undergraduate Computing and Engineering Students in Ireland

Glenn Strong, Catherine Higgins, Nina Bresnihan, Richard Millwood

<figure><img src="image_1.png" /></figure>

## To cite this version:

Glenn Strong, Catherine Higgins, Nina Bresnihan, Richard Millwood. A Survey of the Prior Programming Experience of Undergraduate Computing and Engineering Students in Ireland. 11th IFIP World Conference on Computers in Education (WCCE), Jul 2017, Dublin, Ireland. pp.473-483, 10.1007/978-3-319-74310-3_48. hal-01762850

HAL Id: hal-01762850

https://inria.hal.science/hal-01762850

Submitted on 10 Apr 2018

HAL is a multi-disciplinary open access archive for the deposit and dissemination of scientific research documents, whether they are published or not. The documents may come from teaching and research institutions in France or abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est destinée au dépôt et à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# A Survey of the Prior Programming Experience of Undergraduate Computing and Engineering Students in Ireland

Glenn Strong<fnref n="1" />, Catherine Higgins<fnref n="2" />, Nina Bresnihan<fnref n="1" />, Richard Millwood<fnref n="1" />

{glenn.strong, nina.bresnihan, richard.millwood}@scss.tcd.ie
catherine.higgins@dit.ie

**Abstract.** It has become apparent that increasing numbers of students arriving into undergraduate computing and engineering degree programmes in Irish 3rd-level institutions have prior experience of computer programming. As the extent of this prior exposure as well as its nature, origins, and usefulness is not known beyond anecdotal evidence, an annual survey of prior programming experience of freshman undergraduates who study programming as part of their degree has been designed and administered. This paper reports on the first two years of this survey in 2015 and 2016. It found that around one third had some prior experience of programming with nearly half of that group reporting a reasonable level of fluency in one or more languages. The authors expect that the effect of proposed changes to primary and 2nd-level curricula alongside the increasing popularity of informal programming clubs will be increasingly felt in coming years and therefore plan to continue and extend the survey in order to clarify the effect of such developments. The results should be of interest to 3rd-level educators in the planning of curriculum and teaching practice.

**Keywords:** Computer science education · Programming experience · CS1

## 1. Introduction

In recent years, the academic community in Ireland has become aware that many students arriving at 3rd-level to study computer science and engineering degree programmes do so with some prior exposure to programming. This is despite the fact that programming is not taught as a formal subject in the Leaving Certificate cycle and has only been made available since 2014 as part of an optional Junior Certificate short course in Coding[1]<fnref n="1" />. However, outside of formal educational settings, there has been a surge of interest in learning to program as evidenced by the phenomenal success of the CoderDojo network of after-school programming clubs which since its

<footnote n="1">¹Trinity College Dublin, Ireland</footnote>

<footnote n="2">²Dublin Institute of Technology, Ireland</footnote>

<footnote n="1">¹ Note on the Irish education system: Primary corresponds to K1-K6 in the American system, and Second-level corresponds to K7-K12. The Junior Certificate is the first examination taken at the end of the third year of second-level and the Leaving Certificate is the terminal state exam taken at the end of the K12-equivalent year</footnote>
foundation in 2011 has grown exponentially to include over 200 clubs across Ireland in January 2017[2]. The impact of such initiatives on the intake into 3rd-level courses has yet to be assessed. The standard introductory programming course at 3rd-level assumes no prior knowledge of programming but, in this changing environment, this assumption needs to be questioned. Furthermore, with recent announcements that the National Council for Curriculum and Assessment is to consider approaches to introducing the teaching of programming in primary schools as well as a proposed introduction of Computer Science as a Leaving Certificate subject in 2018 [3], it is clear that 3rd-level institutions will need to adapt to cater for a new generation of students with prior programming experience. In this evolving context, the aim of the survey discussed here is to track the state of programming knowledge among freshman (first year undergraduates) and how this is changing over time thereby providing educators with an evidence-base for decisions about future curriculum design and teaching practice.

The structure of this paper is as follows: Section 2 reports related research while section 3 gives a short overview of the research questions and the process used to collect data to answer these questions. Section 4 reports on the results gathered in the first two years of this survey and section 5 concludes the paper with a discussion of the findings from this research, its future direction and the contribution it makes to the educational research and curriculum development for teaching programming.

## 2. Related Research

A review of related literature revealed that, while prior knowledge of programming among first year undergraduates has been investigated in previous studies, it has generally not been the primary focus of the research. Such studies have investigated the effects of prior knowledge on such things as confidence levels [4], problem solving [5] and gender balance [8] with previous programming experience often being cited as a predictor of programming success [6, 7]. Interestingly, studies have shown that is is not prior knowledge of programming concepts alone but student self-efficacy, or their self-perception of their programming ability that has the strongest correlation with programming performance [9, 10]. Krpan et al's investigation into the correlation between students' success in introductory programming and mathematical courses included an analysis of student expectations and success rates in introductory programming courses at the Faculty of Science, University of Split over three years. They found that early failure to understand basic programming concepts affects students' confidence and leads to increased drop-out rates [11].

The background of Computer Science freshmen across two Finnish universities, was investigated in 2011 and 2012[12]. Differing levels of prior programming experience were found between the two institutions (62% and 38%), with the university with the more selective entrance criteria having the higher percentage. It is worth noting that between 43% and 63% of those with experience gained it through formal studies, an opportunity not currently available to most Irish undergraduates.

Data relating to the prior experience of over 900 students at the Eidgenössische Technische Hochschule (ETH) Zurich was collected over seven years from 2003 and gathered from 77 students from University of York in 2008 [13]. During this time
Computer Science was an optional subject not universally available in either
Switzerland or the UK at primary or 2nd level, yet in both institutions, prior
experience was high, running at a stable level of 78%-84% across the 7 years.
Interestingly, only 16%-25% gained their experience at school, with most reporting
self-study as the source. The authors note that generalisation of their results may be
limited to other universities with similar admission regulations and students who
studied in a comparable school system and recommend that research be broadened to
more universities and countries. Indeed, no research on collating data relating to
students' prior knowledge of programming at a national or international level was
found during our literature search and the survey discussed here is believed to be the
first to attempt to do so on a national level.

3. Research Question and Data Collection

This research project aims to find out what programming knowledge and experience
students have before commencing undergraduate degree programmes with a
programming component in Irish 3rd-level institutions. It also explores how this
exposure affects their experience of programming during their freshman year and how
this picture is changing over time.

To gain answers to these questions, it was decided to undertake a repeated cross-
sectional study using an annual survey. This paper reports on the first two years of
results from that activity. The survey population is freshman students undertaking a
third level, undergraduate degree programme in Ireland which involves the studying
of programming in first year. Such programmes are typically computer science and
engineering programmes. The decision was made to target students in the latter part
of the second semester of their freshman year so they could make a judgement with
regard to the impact of their prior experience on their current studies. While the
overwhelming majority of the survey population would have undergone their second-
level education in Ireland, we note that as of the 2014/2015 academic year
approximately 10% of students in Irish higher education are “international” students
[14]. We expect that a number of these students would have experienced some level
of formal exposure to programming in their previous education.

An online survey was deemed to be the most suitable data collection instrument as
this enabled easy collection of standardised data from a geographically distributed
sample. The benefits of this process were deemed to outweigh our concerns about any
bias resulting from the self-reporting nature of the responses. Ethical approval was
sought and granted by the appropriate ethical committee with Google Forms being the
platform currently used by the researchers to administer and maintain the survey.

Rather than contact the target respondents directly it was decided that a higher
response rate would be likely if they were recruited by teaching staff in their own
institutions. Contact was made with appropriate personnel nationwide to ask for their
collaboration in introducing the goal of the survey and forwarding it to their students.
In return, the researchers furnish each participating institution with their raw student
data as well as a copy of the annual report of analysed results.

In 2015, a pilot study was run between two 3rd-level institutions in Dublin in order
to validate the survey design and gather some baseline data. In this pilot, categories of
questions were developed to understand the profile of the student population in terms of age, gender and programme of study; to ascertain if students had prior experience; to understand the languages learned with the level of fluency and finally to examine the impact of the experience on students’ current studies. The pilot survey was issued to students in both institutions across all 4 years of relevant programmes (students were also asked to identify their year of study as part of the survey, allowing us to make direct comparisons year-by-year). Based on responses to the pilot, the survey was subsequently updated to include two extra categories of questions which examined the source of the student experience as well as their reflections regarding the usefulness of the experience.

From 2016, invitations were issued to 3rd-level institutions nationwide aimed exclusively at freshman students with 8 institutions agreeing to participate which resulted in a sample of n=321 respondents. The aim of the researchers is to continue to grow the institution and participant levels on an annual basis so the generalisability of trends and patterns identified in the years going forward continues to increase. The current design of the survey has 6 categories of questions with a mix of closed multiple choice and open short answer questions producing 32 questions in total. Category 1 examines the profile of the population by asking respondents for personal information such as 3rd-level institution attended, age, gender, and degree programme. Category 2 has a singular focus on asking if respondents have prior experience. For those with experience, there is a further 4 sections. Category 3 examines the nature of their experience in terms of the language(s) learned and the degree of fluency in the language(s). Category 4 examines the origins of their experience from a school, club, online and self-taught perspective. Category 5 examines aspects of their experience that students signal they found particularly helpful in growing their knowledge of programming and finally Category 6 examines how useful students find their prior experience to be in their current freshman studies.

**3.1 Data Validity**

The total population size is only available for two thirds of the group surveyed, which precludes calculating confidence for our overall results, a deficit that will be addressed in future surveys. Where accurate information on cohort size is available, from three institutions (n=222, population=651), we calculate a confidence interval of ±5.34, at a confidence level of 95%. Our overall results do not differ significantly from the results obtained (i.e., they lie within the confidence interval reported) when only considering this restricted set of the population, nevertheless we must be cautious in interpreting the overall results as a result of this weakness.

Some respondents chose not to indicate which institution they were attending, making it impossible to include them in these figures; a requirement of the study's ethical approval was that no question be compulsory making such occasional gaps in the data inevitable. Where it has been possible to uniquely infer a student's institution from their stated course of study we have done so.

With respect to other threats to validity, we note that having students self-assess their level of confidence and ability, rather than applying an objective test, may introduce some bias, though it is not clear in which direction this may lie.
# 4. Results and Analysis

## 4.1 Participants

As the 2015 participants included all 4 years of a programme, we restricted our analysis of the 2015 data to the 122 freshman students, to give a truer basis for comparison. These figures are laid out in Table 1:

<table><caption>Table 1. Participant details</caption><thead><tr><td>Year</td><td>Total</td><td>Male</td><td>Female</td><td>Age 18-22</td><td>Age 22+</td></tr></thead><tbody><tr><td>2015 (n=122)</td><td>122</td><td>72%</td><td>28%</td><td>92%</td><td>8%</td></tr><tr><td>2016 (n=321)</td><td>321</td><td>76%</td><td>24%</td><td>76%</td><td>24%</td></tr></tbody></table>

While only around 25% of respondents are female across both years of the study, this is in fact a slight overrepresentation in the survey as enrolment figures for full-time honours degrees for 2014/2015 for females are a mere 14.9% for computing degrees and 18.1% for Engineering degrees [14]. The reason for the substantial jump in the mature student rate from 2015 to 2016 is unknown.

## 4.2 Programming Experience

The data of most interest was the percentage of students who self-identified as having programming experience. Of the 321 students who responded to the 2016 survey, **66%** indicated that they had **no** prior exposure to programming with **34.2%** indicating they had some experience prior to the start of their 3rd-level course. These figures broadly reflect those from the 2015 pilot survey where **63%** of freshman students (n=120) indicated they had no prior experience. A Chi-squared test showed no significant difference between the two survey results (P=0.938).

Students were also asked to self-report their prior level of fluency in programming. The 2016 cohort reported having a much stronger level of fluency than the 2015 respondents (see figure 1 for figures and an explanation of fluency levels).

A Chi-squared test was performed to confirm the significance (P=0.0176). Reasons for the stronger reported fluency in the 2016 cohort is not apparent from the survey data but it will be interesting to see if this trend continues in the years ahead and to ascertain possible reasons for this increase.

Comparing these results to the results of the ETH Zurich study[13] where 78% - 84% of respondents reported prior experience highlights the relatively low level of experience of Irish students at the outset of their 3rd-level study. We believe this is most likely due to the majority of students coming directly from Irish second-level schools where programming is not yet widely taught. Equally interesting was the level of fluency reported which suggested that for the students who did have experience their previous studies were often deep and considered.
<figure><img src="image_3.png" /><figcaption>Fig. 1. Level of fluency from 2015 and 2016</figcaption></figure>

## 4.3 Programming Languages

Students who reported having some programming experience were then asked to indicate which languages they had some experience with. In 2015 this was presented as multiple choice list (with an “other” option), while in 2016 students were instead presented with an open question giving respondents freedom to indicate the range of languages they had experienced. The two years’ responses are presented in Table 2.

<table><caption>Table 2. Programming languages experienced prior to 3rd-level</caption><thead><tr><td>Language</td><td>2015 freshmen (n=48)</td><td>2016 (n=106)</td></tr></thead><tbody><tr><td>C, C++, C#</td><td>33%</td><td>40%</td></tr><tr><td>HTML</td><td>60%</td><td>28%</td></tr><tr><td>Java</td><td>44%</td><td>59%</td></tr><tr><td>JavaScript</td><td>23%</td><td>17%</td></tr><tr><td>Other</td><td>0%</td><td>22%</td></tr><tr><td>PHP</td><td>0%</td><td>12%</td></tr><tr><td>Python</td><td>25%</td><td>33%</td></tr><tr><td>Scratch</td><td>38%</td><td>1%</td></tr><tr><td>Basic variants</td><td>4%</td><td>14%</td></tr></tbody></table>

Students have clearly engaged with a wide variety of languages which is an indicator of the diversity and range of courses on offer. As an aside, the almost
complete absence of the educational language “Scratch” in the 2016 results is striking. We would speculate that this is due to the change to the open-question format in the 2016 survey causing students to omit Scratch from their answers based on a perception of what constitutes a “real” programming language.

## 4.4 Helpfulness of Experience

Respondents were asked to report on how (or not) their prior programming experience helped them in their freshman year of studying programming. The question was presented as an open-question prompting a varied response. Interestingly, 15% of respondents in 2015 and 10% in 2016 indicated that their prior experience didn't help at all. The analysis of the responses given by the remaining respondents who did report an impact produced the following broad categories:

* A head start boosted confidence helping students transition to 3rd-level.

* Not struggling with the basic concepts allowed for deeper learning to occur.

* Understanding fundamental concepts helped in adapting to new languages.

* More receptive to understanding complex concepts.

* More time could be spent on studying other modules.

As we might expect, prior exposure to programming concepts is considered helpful when dealing with introductory programming modules. Interestingly, the first group that emerged doesn't relate directly to the conceptual knowledge and skills gained from prior experience but instead focuses on the affective impact of having such knowledge particularly in the area of confidence levels and receptiveness to new learning. This underlines the importance of educators adopting pedagogical approaches that support the growth of confidence in learners. Given the diversity of programming tools and skills encountered, it's perhaps not surprising that familiarity with the core concepts of programming had a significant positive impact. Since many introductory programming modules focus on trying to impart these core skills, students who have a head start reported that they can focus purely on advanced constructs or even on other modules.

In 2016 only, respondents were asked, again using an open question, which parts of their prior experience were the most useful to them overall. 35% reported that nothing helped them. This could mean that they either didn't find their experience useful or they were unable to articulate or remember any particular aspects they found useful. The remaining respondents' responses were grouped into the categories listed in figure 2.

<figure><img src="image_4.png" /><figcaption>Fig. 2. Sources of Useful Prior Experience - 2016 (n= 80)</figcaption></figure>
Over a quarter of respondents indicated that while their experience was useful, nothing specific stood out as having helped them. With the remaining responses, the most frequently cited helpful activities were time spent on projects or practice (19%) and specific online or print resources (15%) notably Codeacademy and YouTube.

The fact that a substantial number of participants were unwilling (or unable) to identify individual sources of their experience as being helpful could indicate that nothing at all, or no single specific aspect was helpful. However, alternative readings could suggest that students either had difficulty in reflecting on its effect or that their overall experience - rather than specific experiences - was helpful. This question will need careful redesign in future iterations to assist students answer more accurately without biasing and guiding their responses.

## 4.5 Origins of Experience

In 2016, participants with prior experience were asked a series of questions to gain insight into its origins. These were categorized into school, club, online and self-directed learning with respondents also having an opportunity to indicate any other sources of their experience. This is a particularly interesting question in the Irish context where programming has not been widely available as a formal school subject and grassroots organisations like CoderDojo report considerable activity. Most respondents reported multiple sources for their experience.

**School Activities:** 27% indicated that they had participated in programming activities through school. These activities were identified as short summer classes, transition year courses (a one year programme in Irish second-level schools with no formal exams prior to beginning the two year terminal examination), elective classes, preparation for programming competitions, and as a formal part of school curriculum (international students). The duration of these activities ranged from 1 day to 2 years.

**Club Activities:** 13% of respondents reported participation in clubs or groups outside of school. Of those, 36% had attended CoderDojo; 27% were involved with other computer clubs; 19% were involved with 3rd-level access and youth programmes with the remaining 18% citing adult education courses.

**Online Activities:** Reported participation in online courses (including courses that they may not have finished) was 42%. The most common sites used were Codeacademy (46%), non-specified tutorial sites (27%) and YouTube (7%) with other specified sites each representing just over 2%.

**Self-Directed Learning:** When asked if they had engaged with any self-directed learning such as books or building projects, 63% answered in the affirmative with the majority of those citing projects that they had designed and developed. These included games (41%), websites (22%), web applications (17%), Arduino/Raspberry Pi projects (17%) and general business applications (3%). The time spent on these projects varied significantly from a couple of hours to a couple of years with the average amount reported as approximately 40 hours.

**Other Sources of Experience:** 30% of respondents indicated they had other sources of experience. These included post-2nd-level school courses that were not diplomas or degrees (43%), previous 3rd-level courses (23%), previous employment (13%) with the remaining 21% not specified.

When examining the origins of prior experience, it was not unexpected, given the lack of formal inclusion of programming in Irish schools' curricula, that a high
number (73%) of respondents gained their experience outside of school. Indeed, what was surprising was the extent and range of extra courses run by some schools both as part of their day-to-day curriculum and as extra short courses after school. The percentage of 13% of students participating in clubs/groups would appear to be low when compared to the activity that is school based (27%). This is despite the high profile of CoderDojo but may well be due to the relative infancy of this club, founded in 2011, and we would expect this number to increase in the future in line with the increasing participation and growth of dojos in Ireland. However, this is only speculative analysis and data gathered in the years going forward will allow more concrete patterns of participation in clubs to emerge. At present it is clear that most non-online computing activities undertaken by students are via school.

Not unexpectedly, there is a relatively high number of students undertaking online courses with Codecademy being the clear leader. The most popular languages being learned online are the web development scripting languages and Python. While there is no indicator in the survey regarding the reasons why students chose those languages, having an understanding of popular online programming paradigms could provide assistance both in understanding prior experience and also in the generation of online courses for distance education at 3rd-level.

# 5. Discussion

When examining the profile of freshman students in any discipline, there is always diversity in terms of age, background and general experience. This is a factor that 3rd-level educators have always had to take into consideration when designing courses and choosing appropriate pedagogical approaches. With computer science primary degrees having the highest rate of non-progression in Ireland (varying between universities (15%) and institutes of technology (26%)) [15]) it is clearly important that the nature and impact of that previous experience be more fully understood so that introductory courses be made more relevant and appealing to students.

While Ireland has not as yet introduced formal computer science or programming as mainstream subjects in schools, it has become apparent to educators that a growing number of students are already presenting in Irish 3rd-level institutions with prior programming experience. From the results of the survey presented in this paper, it can be seen that roughly a third of those surveyed have had some exposure to programming. To date, a lot of that experience is, necessarily, from self-directed and online study but a surprising amount stems from schools which emerged as an important driver and facilitator for encouraging students into their first steps in programming. Non-school clubs such as CoderDojo have grown in numbers and popularity since their foundation in 2011, however their effect in Ireland cannot be seen as yet in the 3rd-level population. Together with the mooted introduction of programming through the new mathematics curriculum at primary level and the introduction of computer science as a Leaving Certificate subject from September 2018, it is clear that the nature and extent of the prior programming experience of incoming undergraduates will be subject to enormous change over the coming decade.

The early results of this study, while limited in their scope, can provide an indication of the current situation as well as a baseline from which to evaluate the
inevitable further changes in our student profile. The authors will continue to expand
the reach of this survey and administer it annually in order to track these changes and
see how they impact student interest and experience in programming going forward.
Given the shortage of comparable national studies from other countries, it is difficult
to compare these results against international standards. However, given the growing
interest in computer science and programming at school level worldwide and the
resulting increase in the skills and knowledge of incoming 3rd-level students, it is
expected that research and interest in this area will grow. We would argue that this
survey will lead to a better understanding of student experience prior to their arrival at
3rd-level and help to provide an evidence-base for decisions about future curriculum
design and teaching practice, consequently providing some contribution to student
retention and competency levels.

**Acknowledgments.** The authors would like to thank our colleagues at various institutions in Ireland who have circulated the survey to their students.

References

1. National Council for Curriculum and Assessment: JuniorCycle.ie - Short Courses, http://www.juniorcycle.ie/Curriculum/Short-Courses.

2. N.A.: CoderDojo Zen, https://zen.coderdojo.com/.

3. Department of Education and Skills: Action Plan for Education 2017 (2017)

4. Macleod, H., Patterson, J.: A survey of undergraduate technology use and attitudes. University of Edinburgh. https://doi.org/10.7488/ds/97 (2012)

5. Chen, T.-Y., Lewandowski, G., McCartney, R., Sanders, K., Simon, B.: Commonsense computing: using student sorting abilities to improve instruction. In: ACM SIGCSE Bulletin, pp. 276–280. ACM (2007)

6. Madigan, E.M., Goodfellow, M., Stone, J.A.: Gender, perceptions, and reality: technological literacy among first-year students. Presented at the ACM SIGCSE Bulletin (2007)

7. Hagan, D., Markham, S.: Does It Help to Have Some Programming Experience Before Beginning a Computing Degree Program? In: Proceedings of the 5th Annual SIGCSE/SIGCUE ITiCSEconference on Innovation and Technology in Computer Science Education, pp. 25–28. ACM, New York, NY, USA (2000)

8. Cantwell Wilson, B.: A Study of Factors Promoting Success in Computer Science Including Gender Differences. Computer Science Education. 12, 141–164 (2002)

9. Bergin, S., Reilly, R. 2005. Programming: factors that influence success. In: ACM SIGCSE Bulletin, 37(1): pp. 411–415.

10. Cantwell Wilson, B., Shrock, S. 2001. Contributing to success in an introductory computer science course: a study of twelve factors. ACM SIGCSE Bulletin, 33(1): pp. 184–188.

11. Krpan, D., Mladenović, S., Rosić, M.: Undergraduate programming courses, students’ perception and success. In: Procedia-Social and Behavioral Sciences 174 (2015): 3868-3872.

12. Kinnunen, P., Marttila-Kontio, M., Pesonen, E.: Getting to Know Computer Science Freshmen. In: Proceedings of the 13th Koli Calling International Conference on Computing Education Research, pp. 59–66. ACM, New York, NY, USA (2013)

13. Pedroni, M., Oriol, M., Meyer, B.: The prior experience of entering CS students. ETH, Department of Computer Science (2011)

14. Higher Education Authority: Student Enrollment Statistics 2014-15, http://www.hea.ie/node/1557.

15. Higher Education Authority: A Study of Progression in Irish Higher Education 2012/13 to 2013/14 (2016)