Using CTFs for an Undergraduate Cyber Education

Over the last five years, the United States Air Force Academy (USAFA) has participated in numerous Capture the Flag (CTF) and other cyber competitions. At first, this was simply an extracurricular club activity; however, as we have seen the impact on student motivation and learning, we have greatly increased student and faculty participation. Additionally, we have started to base entire for-credit courses on a CTF framework. In this paper we discuss our rationale for utilizing CTFs as part of our formal curriculum, as well as key lessons learned relating to student engagement and avoiding cribbing. Introduction Students at the United States Air Force Academy (USAFA) started competing in Capture the Flag (CTF) events for fun, without any formal organization or faculty involvement. In December 2010, we formally created the USAFA Cyber Competition Team. This team competes in international Capture-The-Flag (CTF) events held at most major cyber conferences, as well as in cyber defense competitions (e.g. the National Collegiate Cyber Defense Competition). Through our experience in fielding a cyber team, we have grown to appreciate students’ significant learning from these competitions. Most notable is the increased motivation level of some students. Students would put in many hours trying to solve a CTF challenge, doing research on the web and implementing possible solutions. Their motivation was significantly greater than for their traditional class assignments. The learning also helped them with their regular coursework. Given these observations, over the last few years we have experimented with an expanded version of this approach by taking our cyber-related curriculum and converting it into a CTF-style classroom. In this paper, we discuss the type of curriculum we have folded into a CTF framework and how we have applied it in academic, training, and extra-curricular activities. We then discuss the impact that the CTF-driven curriculum has on our students. Finally, we discuss some of the challenges we faced with this approach and provide mechanisms that other schools can use to implement a CTF style classroom. Encouraging Cyber at USAFA At USAFA we have taken several complementary approaches to delivering Cyber content to the students. As a military service academy, we feel that all of our students should have a basic exposure to cyber concepts primarily from a defensive perspective. We accomplish this through fifteen lessons in our core computing course, which all freshmen are required to take. In this course, we provide several laboratory activities that provide hands-on experiences to help students understand the inherent risks and mitigations of the cyber environment. These activities are focused on a few fundamental topics including social engineering, malware, and the high-level vulnerabilities to national infrastructure. We approach each of the lessons with hands-on labs to understand what the purpose of the attack is, how the attacker would operate, and how the victim should respond. Following the core course we have a Cyber Training elective. In this elective approximately 17% of the rising sophomores are exposed to cyber topics through a CTF architecture, CyberStakes Online, provided by ForAllSecure and funded by DARPA. This is a student-run program where the rising juniors and seniors provide mentoring and support while the sophomores work through the challenges. The focus of this training program is to provide a more detailed understanding of the different facets of the Cyber domain. In this CTF architecture, students are provided problems that have been categorized as basic tutorial, forensics, binary, reversing, web, or crypto. Under each of these categories the challenges are organized with points that increase consistently with the difficulty of the problems. With this structure students are given the flexibility to move through the categories they most want to learn about. This CTF approach provides us with several incentives that we use to encourage participation. First, students earn points which they can compare against other students as an ongoing competition (Figure 1). Second, we have created a local incentive program whereby students who correctly answer all questions over a threshold while accumulating an overall score over another threshold can earn their US Air Force Academy Cyber Wings. These students are then able to wear these wings on their uniforms as an official symbol of their accomplishment (see Figure 2). Figure 1: Scoreboard of our CTF environment Figure 2: Cadet Basic Cyber Badge Students who want to continue their cyber education can declare the Computer and Network Security major, join the Cadet Cyber Competition Team, and/or volunteer to lead the summer Cyber Training courses. Computer and Network Security majors will take several in-depth courses on cyber related technical and policy/legal related topics. Those who join the Cyber Competition Team compete internationally against undergraduate, graduate, and professional organizations in capture the flag, vulnerable-box, and network defense challenges. Finally, those who elect to serve as leadership of the Cyber Training program continue to work on our in-house CTF with the goal of enhancing and expanding the learning of subsequent offerings. Gaming the curriculum Bringing CTFs into the classroom continues our tradition of innovation to make the classroom more interactive and accessible. In 2004, we created a new flowchart-based programming environment, RAPTOR, to abstract away typical programming syntax and make algorithmic thinking more accessible to non-majors (Carlisle et. al 2004). USAFA computer science faculty have also developed videos (Bower, 2010, Carlisle, et al. 2010) to invert the classroom, allowing more time in class to explore and answer questions, moving more traditional lecture outside the classroom. More recently, while at USAFA, Adrian de Freitas brought gaming into the lecture hall through an interactive quiz experience, Classroom Live (de Freitas et al 2013). In Classroom Live, students are able to upgrade and progress their avatars through correctly answering questions. With our Cyber curriculum, we have expanded on the inverted classroom approach to provide activities for students that range from largescale Jeopardy style activities to multi-day exercise style events with external partners. We use the large scale exercises as culminating activities where students are matched against external partners for direct head-to-head competition. In the cyber curriculum these competitions are constructed to offer our students an ability to act as both cyber defenders and cyber attackers. In a defensive event, our students act as a blue team who are defending their electronic resources (network, data, etc.) from a competing organization, acting as red-team, who is actively engaging in some type of attacks to steal/break the resource. In the offensive exercise the roles are reversed. For either event we typically pair an experienced student with the team to coach them on what they should be looking for and what options they can take to respond. These events provide students a strong sense of realism and purpose for what they have learned leading into the events but they are typically very difficult to set up and are only useful in reinforcing concepts that they have mastered in previous, less all-inclusive, activities. To teach the individual concepts of the cyber curriculum we have adopted CyberStakes Online, a DARPA-funded CTF architecture created by the authors of PicoCTF (Chapman et al 2014). The CTF structure allows students to work through the learning objectives at their own pace. Through its scoring system, they are each able to compete against themselves and also can see how they rank amongst their peers. This structure has enabled us to provide curriculum well beyond what we were previously able to accomplish in a more traditional lecture or even inverted classroom. Since students are able to progress at their own pace, and they have the sense of competition, we have found that the challenges themselves have provided sufficient motivation for most of the students to progress through the material. Hints provided with each of the challenges help students figure out what to research on the web to find a solution. We also include team-based activities. These team activities span three genres: defensive exercises, offensive exercises, and individual-based cumulative scoring exercises. The last of those is essentially using the CyberStakes framework where groups of individuals are motivated to help each other learn by pitting small teams against each other and comparing the cumulative score of each team. This type of team exercise is easy to facilitate compared to defensive and offensive exercises where each team member has specific roles and must therefore operate together to achieve an objective. Within the context of the curriculum, USAFA participates in one defense-only competitive exercise, the NSA’s interservice Cyber Defense Exercise (CDX). (The USAFA Competition Team also participates in the National Collegiate Cyber Defense Competition). The CDX strives to place students in the role of network builders and operators fending off malicious attacks from external aggressors and naive or malicious users. This exercise is by far the most complex competition we do each year. Large amounts of time are spent by both the NSA and the student teams on developing and testing scoring systems, establishing and interpreting rules for red and blue team activities, building custom networks, and finding and training “white” cell staff to mediate the exercise. The plus side of this exercise is that it encompasses a wide array of student engagement from building, analyzing and assessing through defending a network. PPP (Plaid 2014) notes several downsides to defense-only exercises. The one that resonates with us the mo