# The Utilization of the iOS Platform to Create LearnENT:  An Interactive Educational App in Otolaryngology—Head and  Neck Surgery

Authors: S. Kohlert; N. Scherer; S. Kherani; L. McLean
Journal: Education Research International
 (2012)

---

## Abstract

LearnENT, an educational app for iOS, was developed to promote a standardized experience otolaryngology in head and neck surgery (OTOHNS) for University of Ottawa medical students. Its development was grounded in pedagogical theory including Laurillard’s design process, Honey and Mumford’s learning styles, and Nielsen’s theory of usability. This paper examines LearnENT's design and development processes as well as the role of mobile apps in medical education. Features of the LearnENT app as they apply to Constructivist learning are also highlighted.


---

## Body

## 1. Introduction

The Constructivist Theory of Learning (CTL) states that learning is improved when the learner is engaged, the material is relevant to the task, and the learner chooses how and when to learn [1]. Electronic learning (e-learning) has been shown to be a viable modality to address CTL when pedagogically sound instructional design is utilized.Undergraduate medical education in otolaryngology and head and neck Surgery (OTOHNS) is highly variable [2] in Canadian medical schools. At the University of Ottawa, the duration of the undergraduate clerkship rotation in otolaryngology is one week. To help provide a standardized OTOHNS exposure and to emphasize key objectives to students in their clerkship at the University of Ottawa, LearnENT, a mobile (Apple iOS-based) application was developed.This paper will focus on the pedagogical and technological aspects of the development of LearnENT and the reasons behind several of the app’s major features.


## 2. Methods

The development of LearnENT was guided by Laurillard’s learning design process [3, 4], (Figure 1).Figure 1
Laurillard’s learning design process.(1) Determination of Blueprint
Campisi describes five core clerkship goals for OTOHNS (Table1) [2]. Combined with a review of the established OTOHNS clerkship objectives at the University of Ottawa, two primary goals for the LearnENT app were established:(a)
provide the student with the ability to learn/review the pertinent OTOHNS physical exam skills prior to beginning their clinical rotation in Otolaryngology (allowing them to gain the most of their experience during this short rotation);(b)
expose the student to interactive clinical cases representing common and/or life threatening conditions in OTOHNS to ensure a standardized exposure to these conditions for all students.Table 1
Core clerkship objectives (from Campisi et al.).
Core clerkship goalsDemonstrate the ability to obtain a complete otolaryngology history.Be able to perform a complete otolaryngology physical examination and be able to arrive at a tentative diagnosis.Be able to differentiate between life-threatening, serious, and minor conditions.Recognize the necessity to refer to otolaryngology specialists.Be conversant with various investigative and treatment modalities used in otolaryngology.(2) Choice of Platform
The Apple iOS platform (iPhone, iPad, and iPod Touch) was chosen primarily because it has been widely adopted throughout the world; with more than 200 million devices sold, it is the leading mobile operating system [5]. The medical community has also embraced iOS: a 2011 survey of 2,041 American physicians reported that 75% owned at least one iOS device, while 28% planned to purchase an iPad within 6 months [6, 7]. Other benefits to iOS include its mature software development kit, flexible language [8], high degree of interactivity, and relative lack of platform fragmentation [9, 10].(3) Coding
The iOS application was coded in the Objective-C language using Apple Xcode 4.2(4) Feature Development
Content was formatted into text, illustrations, photos, and video. Special features include a “My Notes” section, interactive case studies, and a credit-based investigations section that encourages responsible utilization of resources. Many features will be discussed in detail below.(5) Testing
Beta testing was performed by the authors and several colleagues. One focus group session (composed of 20 first and second year medical students at the University of Ottawa) was performed to assess the students’ needs as well as the app’s core functionality and usability. A near-compete version of the app was then reviewed for content errors by the three senior authors before deployment.(6) Deploying
The app will be distributed free of charge on the iOS App Store.(7) Evaluating
Usage data will be collected anonymously using preexisting APIs and stored on a central server. User opinion will be gauged through an in-app evaluation. External peer-review will be sought. Follow-up studies will be performed to assess user experience and learning outcomes. This assessment process will be discussed in detail below.


## 3. Discussion

### 3.1. Benefits of eLearning

eLearning has become extremely prevalent in North American Medical Schools: a recent survey found that 97% offers online course materials [11]. Many institutions, including the University of Ottawa, have enjoyed significant success employing tools such as podcasts for undergraduate medical instruction. Using pre- and post- tests to assess performance, Barrett and colleagues showed that students who listened to a single podcast about cardiac murmurs showed dramatic improvement in heart sound recognition [12].


### 3.2. The Evolution of eLearning: From Desktop to Mobile

Online learning is a portable tool that allows the student to meet the elements of the CTL. However, accessibility of online learning can be compromised in locations where learners do not have internet connectivity, for example during a daily commute. A 2008 survey of medical residents at Dalhousie University found that 89% owned a personal digital assistant (PDA) [13]. Similarly, smartphone sales have grown exponentially since 2008, [14, 15] (Figure 2). Additionally, of the 70% of physicians who own a smartphone, 95% downloads medical applications (apps) [16]. Due to the high penetration of smartphones in the medical world, it was appropriate to publish our eLearning modules as mobile app. While over 425,000 apps [5] (including 5,000 medical apps) have been published to the Apple App Store to date, not one has been geared towards medical professionals and trainees in OTOHNS.Figure 2
Worldwide smartphone sales (in millions).


### 3.3. User Experience

Nielsen described the 5 attributes of usability as beinglearnability (the user should be able to reach “a reasonable level of usage proficiency within a short time”), efficiency (“once the user has learned the system, a high level of productivity [should be] possible”), memorability (the user should remember how to use the app “based on previous learning”), errors (the user should make few and “noncatastrophic” errors and should be able to easily recover from any errors they do make), and satisfaction (the end user should be satisfied with the application and its functions) [17]. Table 2 illustrates how LearnENT applies Nielsen's core principles of usability.Table 2
LearnENT’s approach to Nielsen’s attributes of learnability.
Learnability & MemorabilityOur UI is unique but acts similarly to iOS’ standard UI, making it immediately familiar. All custom elements are introduced and explained the first time the user encounters them.EfficiencyAll features can be accessed from the home page. Each module can be done a few minutes at a time w/o losing the user's progress.ErrorsAuto-save & auto-backup (through iTunes/iCloud). Undo/Redo capability when entering text.SatisfactionLearnENT is a living entity. Feedback received through in-app evaluation and reviews will be used for improving the app (free updates).UI: User Interface.


### 3.4. Educational Design

In order to develop any form of pedagogical tool, an understanding of learning theory and how it can be applied to one’s field is paramount. This section will discuss basic pedagogical theory as it pertains to LearnENT.


#### 3.4.1. Learning Styles

Honey and Mumford developed a learning theory that describes 4 learning styles: the reflector, the theorist, the pragmatist, and the activist (Appendix A) [18, 19]. By using multiple teaching modalities, LearnENT attempts to cater to learners of any style. For the more deliberate/reflective learners (i.e., “reflectors” & “theorists”) LearnENT provides directed, step-by-step learning modules that can be completed at the learner’s preferred pace. A detailed list of recommended references is also provided at the end of each module. For those who learn better through active experimentation (i.e., “activists” & “pragmatists”) LearnENT offers features such as video tutorials (Figure 3), interactive activities and case-based learning modules.Figure 3
Video examples allow visual learners to develop their physical examination skills.


#### 3.4.2. Universal Principles

Although LearnENT makes use of several different teaching modalities, each section of the app was developed in keeping with the following guiding principles. First, training modules should not be static, as efficient learning requires practice, feedback, and guidance [20]. Learners are also more likely to be motivated and engaged by training that is challenging and goal oriented [21]. Perhaps most importantly, LearnENT offers a highly interactive user experience (Figures 4 and 5(b)). A high degree of interactivity not only engages the user but also results in deeper learning as it allows the user to “hypothesize to test their understanding, learn by mistakes and make sense of the unexpected” [22].Figure 4
Interactive quizzes provide instantaneous feedback to the user.(a) Part of a short, interactive tutorial that automatically appears the first time the app is launched. (b) clear instructions for one highly interactive quiz.
(a)(b)


#### 3.4.3. Case Simulations

Case-based simulations foster critical thinking [23], decision making skills [24], and clinical confidence [25]. LearnENT offers realistic, interactive cases that allow the user to develop clinical reasoning skills in a safe environment.


#### 3.4.4. Ease of Use

In order to quickly introduce users to LearnENT’s various features, visually pleasing graphic overlays were designed (Figures5(a) and 5(b)).Interspersed within the app are short, relevant blocks of content that may serve to increase learner productivity and knowledge retention. In each physical exam section, helpful suggestions are highlighted through the use of easily identifiable tip boxes (Figure7(a)).These overlays are context dependent and appear the first time users access a novel feature. They are designed to visually introduce the user to the feature in a manner that is easy to understand and does not force them to read paragraphs of explanatory text. Due to the nature of context-dependent help screens, the user is no longer tethered to a printed instructional manual and is automatically presented with the appropriate help screen whenever applicable.


#### 3.4.5. Quick Access to Reliable, Relevant Information

LearnENT uses a simple and organized user interface to provide quick and easy access to the information within the app. Users with specific queries may use LearnENT’s app-wide search option to locate all sections containing a certain keyword (Figure6).Figure 6
Easily search for key topics using LearnENT’s custom search engine.Tip boxes (a) And emergency boxes (b) Stand out from the text and highlight important clinical pearls.
(a)(b)Similarly, urgent clinical information is emphasized through the use of red emergency boxes (Figure7(b)). In each anatomy review section, short clinical vignettes help to correlate the basic anatomy with the pathophysiology of common OTOHNS diseases. Finally, each section also features a short list of common mistakes that new learners may make while performing the physical exam (Figure 8).Figure 8
Common mistake sections highlight common learner challenges for each facet of the OTOHNS physical exam.To further facilitate learning, users can easily look up the definition of most words using the built-in dictionary as shown in Figure9. By allowing the user to define unfamiliar words without leaving LearnENT, we not only create a more immersive and engaging experience for the user but also save the user valuable time in the process.Figure 9
Built-in dictionary allows the user to quickly define unfamiliar words terminology without the hassle of leaving the app.


#### 3.4.6. “My Notes”

The process of note taking has been repeatedly shown to increase the effectiveness and transfer of learning [26, 27] and results in better academic achievement [28]. LearnENT’s novel “My Notes” feature (Figure 10) can be immediately accessed from nearly every page in the app, allowing learners to easily take notes without losing their progress. Furthermore, all notes can be effortlessly printed or emailed without having to leave the app.Figure 10
“My Notes” section allows users to save, print, and email notes without exiting the app.


#### 3.4.7. “Investigations”

Diagnostic tests are often overused in modern medicine: one study found that “65% of laboratory tests, 11% of CXRs, and 26% of nursing services could not be justified” [29]. This may be a function of physicians’ limited knowledge of diagnostic test cost. Innes and colleagues showed that the mean error of ED physicians’ cost estimates was 40% for imaging studies and 153% for laboratory tests [30] while a separate study found that posting the cost at point of care reduced test utilization by 10–15% [31].Every case-based module in LearnENT features an innovative “Investigations” section (Figure11) that was designed to encourage careful and appropriate use of such tests. Each test has been assigned a fictional value (in “credits”) that is determined based on its real world cost, risk, and availability. For added realism, a large assortment of tests is available in each case, many of them unnecessary for the current patient. The learner’s decisions are recorded and specific, targeted feedback is given at the end of each case.Figure 11
“Investigations” sections promote careful and appropriate use of diagnostic tests.


## 3.1. Benefits of eLearning

eLearning has become extremely prevalent in North American Medical Schools: a recent survey found that 97% offers online course materials [11]. Many institutions, including the University of Ottawa, have enjoyed significant success employing tools such as podcasts for undergraduate medical instruction. Using pre- and post- tests to assess performance, Barrett and colleagues showed that students who listened to a single podcast about cardiac murmurs showed dramatic improvement in heart sound recognition [12].


## 3.2. The Evolution of eLearning: From Desktop to Mobile

Online learning is a portable tool that allows the student to meet the elements of the CTL. However, accessibility of online learning can be compromised in locations where learners do not have internet connectivity, for example during a daily commute. A 2008 survey of medical residents at Dalhousie University found that 89% owned a personal digital assistant (PDA) [13]. Similarly, smartphone sales have grown exponentially since 2008, [14, 15] (Figure 2). Additionally, of the 70% of physicians who own a smartphone, 95% downloads medical applications (apps) [16]. Due to the high penetration of smartphones in the medical world, it was appropriate to publish our eLearning modules as mobile app. While over 425,000 apps [5] (including 5,000 medical apps) have been published to the Apple App Store to date, not one has been geared towards medical professionals and trainees in OTOHNS.Figure 2
Worldwide smartphone sales (in millions).


## 3.3. User Experience

Nielsen described the 5 attributes of usability as beinglearnability (the user should be able to reach “a reasonable level of usage proficiency within a short time”), efficiency (“once the user has learned the system, a high level of productivity [should be] possible”), memorability (the user should remember how to use the app “based on previous learning”), errors (the user should make few and “noncatastrophic” errors and should be able to easily recover from any errors they do make), and satisfaction (the end user should be satisfied with the application and its functions) [17]. Table 2 illustrates how LearnENT applies Nielsen's core principles of usability.Table 2
LearnENT’s approach to Nielsen’s attributes of learnability.
Learnability & MemorabilityOur UI is unique but acts similarly to iOS’ standard UI, making it immediately familiar. All custom elements are introduced and explained the first time the user encounters them.EfficiencyAll features can be accessed from the home page. Each module can be done a few minutes at a time w/o losing the user's progress.ErrorsAuto-save & auto-backup (through iTunes/iCloud). Undo/Redo capability when entering text.SatisfactionLearnENT is a living entity. Feedback received through in-app evaluation and reviews will be used for improving the app (free updates).UI: User Interface.


## 3.4. Educational Design

In order to develop any form of pedagogical tool, an understanding of learning theory and how it can be applied to one’s field is paramount. This section will discuss basic pedagogical theory as it pertains to LearnENT.


### 3.4.1. Learning Styles

Honey and Mumford developed a learning theory that describes 4 learning styles: the reflector, the theorist, the pragmatist, and the activist (Appendix A) [18, 19]. By using multiple teaching modalities, LearnENT attempts to cater to learners of any style. For the more deliberate/reflective learners (i.e., “reflectors” & “theorists”) LearnENT provides directed, step-by-step learning modules that can be completed at the learner’s preferred pace. A detailed list of recommended references is also provided at the end of each module. For those who learn better through active experimentation (i.e., “activists” & “pragmatists”) LearnENT offers features such as video tutorials (Figure 3), interactive activities and case-based learning modules.Figure 3
Video examples allow visual learners to develop their physical examination skills.


### 3.4.2. Universal Principles

Although LearnENT makes use of several different teaching modalities, each section of the app was developed in keeping with the following guiding principles. First, training modules should not be static, as efficient learning requires practice, feedback, and guidance [20]. Learners are also more likely to be motivated and engaged by training that is challenging and goal oriented [21]. Perhaps most importantly, LearnENT offers a highly interactive user experience (Figures 4 and 5(b)). A high degree of interactivity not only engages the user but also results in deeper learning as it allows the user to “hypothesize to test their understanding, learn by mistakes and make sense of the unexpected” [22].Figure 4
Interactive quizzes provide instantaneous feedback to the user.(a) Part of a short, interactive tutorial that automatically appears the first time the app is launched. (b) clear instructions for one highly interactive quiz.
(a)(b)


### 3.4.3. Case Simulations

Case-based simulations foster critical thinking [23], decision making skills [24], and clinical confidence [25]. LearnENT offers realistic, interactive cases that allow the user to develop clinical reasoning skills in a safe environment.


### 3.4.4. Ease of Use

In order to quickly introduce users to LearnENT’s various features, visually pleasing graphic overlays were designed (Figures5(a) and 5(b)).Interspersed within the app are short, relevant blocks of content that may serve to increase learner productivity and knowledge retention. In each physical exam section, helpful suggestions are highlighted through the use of easily identifiable tip boxes (Figure7(a)).These overlays are context dependent and appear the first time users access a novel feature. They are designed to visually introduce the user to the feature in a manner that is easy to understand and does not force them to read paragraphs of explanatory text. Due to the nature of context-dependent help screens, the user is no longer tethered to a printed instructional manual and is automatically presented with the appropriate help screen whenever applicable.


### 3.4.5. Quick Access to Reliable, Relevant Information

LearnENT uses a simple and organized user interface to provide quick and easy access to the information within the app. Users with specific queries may use LearnENT’s app-wide search option to locate all sections containing a certain keyword (Figure6).Figure 6
Easily search for key topics using LearnENT’s custom search engine.Tip boxes (a) And emergency boxes (b) Stand out from the text and highlight important clinical pearls.
(a)(b)Similarly, urgent clinical information is emphasized through the use of red emergency boxes (Figure7(b)). In each anatomy review section, short clinical vignettes help to correlate the basic anatomy with the pathophysiology of common OTOHNS diseases. Finally, each section also features a short list of common mistakes that new learners may make while performing the physical exam (Figure 8).Figure 8
Common mistake sections highlight common learner challenges for each facet of the OTOHNS physical exam.To further facilitate learning, users can easily look up the definition of most words using the built-in dictionary as shown in Figure9. By allowing the user to define unfamiliar words without leaving LearnENT, we not only create a more immersive and engaging experience for the user but also save the user valuable time in the process.Figure 9
Built-in dictionary allows the user to quickly define unfamiliar words terminology without the hassle of leaving the app.


### 3.4.6. “My Notes”

The process of note taking has been repeatedly shown to increase the effectiveness and transfer of learning [26, 27] and results in better academic achievement [28]. LearnENT’s novel “My Notes” feature (Figure 10) can be immediately accessed from nearly every page in the app, allowing learners to easily take notes without losing their progress. Furthermore, all notes can be effortlessly printed or emailed without having to leave the app.Figure 10
“My Notes” section allows users to save, print, and email notes without exiting the app.


### 3.4.7. “Investigations”

Diagnostic tests are often overused in modern medicine: one study found that “65% of laboratory tests, 11% of CXRs, and 26% of nursing services could not be justified” [29]. This may be a function of physicians’ limited knowledge of diagnostic test cost. Innes and colleagues showed that the mean error of ED physicians’ cost estimates was 40% for imaging studies and 153% for laboratory tests [30] while a separate study found that posting the cost at point of care reduced test utilization by 10–15% [31].Every case-based module in LearnENT features an innovative “Investigations” section (Figure11) that was designed to encourage careful and appropriate use of such tests. Each test has been assigned a fictional value (in “credits”) that is determined based on its real world cost, risk, and availability. For added realism, a large assortment of tests is available in each case, many of them unnecessary for the current patient. The learner’s decisions are recorded and specific, targeted feedback is given at the end of each case.Figure 11
“Investigations” sections promote careful and appropriate use of diagnostic tests.


## 3.4.1. Learning Styles

Honey and Mumford developed a learning theory that describes 4 learning styles: the reflector, the theorist, the pragmatist, and the activist (Appendix A) [18, 19]. By using multiple teaching modalities, LearnENT attempts to cater to learners of any style. For the more deliberate/reflective learners (i.e., “reflectors” & “theorists”) LearnENT provides directed, step-by-step learning modules that can be completed at the learner’s preferred pace. A detailed list of recommended references is also provided at the end of each module. For those who learn better through active experimentation (i.e., “activists” & “pragmatists”) LearnENT offers features such as video tutorials (Figure 3), interactive activities and case-based learning modules.Figure 3
Video examples allow visual learners to develop their physical examination skills.


## 3.4.2. Universal Principles

Although LearnENT makes use of several different teaching modalities, each section of the app was developed in keeping with the following guiding principles. First, training modules should not be static, as efficient learning requires practice, feedback, and guidance [20]. Learners are also more likely to be motivated and engaged by training that is challenging and goal oriented [21]. Perhaps most importantly, LearnENT offers a highly interactive user experience (Figures 4 and 5(b)). A high degree of interactivity not only engages the user but also results in deeper learning as it allows the user to “hypothesize to test their understanding, learn by mistakes and make sense of the unexpected” [22].Figure 4
Interactive quizzes provide instantaneous feedback to the user.(a) Part of a short, interactive tutorial that automatically appears the first time the app is launched. (b) clear instructions for one highly interactive quiz.
(a)(b)


## 3.4.3. Case Simulations

Case-based simulations foster critical thinking [23], decision making skills [24], and clinical confidence [25]. LearnENT offers realistic, interactive cases that allow the user to develop clinical reasoning skills in a safe environment.


## 3.4.4. Ease of Use

In order to quickly introduce users to LearnENT’s various features, visually pleasing graphic overlays were designed (Figures5(a) and 5(b)).Interspersed within the app are short, relevant blocks of content that may serve to increase learner productivity and knowledge retention. In each physical exam section, helpful suggestions are highlighted through the use of easily identifiable tip boxes (Figure7(a)).These overlays are context dependent and appear the first time users access a novel feature. They are designed to visually introduce the user to the feature in a manner that is easy to understand and does not force them to read paragraphs of explanatory text. Due to the nature of context-dependent help screens, the user is no longer tethered to a printed instructional manual and is automatically presented with the appropriate help screen whenever applicable.


## 3.4.5. Quick Access to Reliable, Relevant Information

LearnENT uses a simple and organized user interface to provide quick and easy access to the information within the app. Users with specific queries may use LearnENT’s app-wide search option to locate all sections containing a certain keyword (Figure6).Figure 6
Easily search for key topics using LearnENT’s custom search engine.Tip boxes (a) And emergency boxes (b) Stand out from the text and highlight important clinical pearls.
(a)(b)Similarly, urgent clinical information is emphasized through the use of red emergency boxes (Figure7(b)). In each anatomy review section, short clinical vignettes help to correlate the basic anatomy with the pathophysiology of common OTOHNS diseases. Finally, each section also features a short list of common mistakes that new learners may make while performing the physical exam (Figure 8).Figure 8
Common mistake sections highlight common learner challenges for each facet of the OTOHNS physical exam.To further facilitate learning, users can easily look up the definition of most words using the built-in dictionary as shown in Figure9. By allowing the user to define unfamiliar words without leaving LearnENT, we not only create a more immersive and engaging experience for the user but also save the user valuable time in the process.Figure 9
Built-in dictionary allows the user to quickly define unfamiliar words terminology without the hassle of leaving the app.


## 3.4.6. “My Notes”

The process of note taking has been repeatedly shown to increase the effectiveness and transfer of learning [26, 27] and results in better academic achievement [28]. LearnENT’s novel “My Notes” feature (Figure 10) can be immediately accessed from nearly every page in the app, allowing learners to easily take notes without losing their progress. Furthermore, all notes can be effortlessly printed or emailed without having to leave the app.Figure 10
“My Notes” section allows users to save, print, and email notes without exiting the app.


## 3.4.7. “Investigations”

Diagnostic tests are often overused in modern medicine: one study found that “65% of laboratory tests, 11% of CXRs, and 26% of nursing services could not be justified” [29]. This may be a function of physicians’ limited knowledge of diagnostic test cost. Innes and colleagues showed that the mean error of ED physicians’ cost estimates was 40% for imaging studies and 153% for laboratory tests [30] while a separate study found that posting the cost at point of care reduced test utilization by 10–15% [31].Every case-based module in LearnENT features an innovative “Investigations” section (Figure11) that was designed to encourage careful and appropriate use of such tests. Each test has been assigned a fictional value (in “credits”) that is determined based on its real world cost, risk, and availability. For added realism, a large assortment of tests is available in each case, many of them unnecessary for the current patient. The learner’s decisions are recorded and specific, targeted feedback is given at the end of each case.Figure 11
“Investigations” sections promote careful and appropriate use of diagnostic tests.


## 4. Future Directions

### 4.1. Objective Assessment

Although LearnENT has already undergone significant beta testing, the next phase involves more extensive review. Evaluating how, when, where and why students use the app will be paramount as we attempt to elucidate the role of future apps in the educational realm [32].Using existing technologies such as TestFlight and Flurry Analytics, we are able to develop a valuable understanding of how users interact with LearnENT. First, we will determine adoption and retention rates using statistics such as number of new and returning users, as well as session length and frequency of use. As it is free and easily available, we expect LearnENT to have a high adoption rate. We will be more interested; however, in the retention rate, as well as session frequency and length which will allow us to infer the users’ opinions of the app’s value. Additionally, we will track each section individually, allowing us to determine which sections are the most and least popular. This data will potentially allow us to extrapolate the preferred learning styles of our users, which will be used in the development of future material. Finally, user demographics such as country of origin can be used to create targeted content, including multilingual versions of the app. To allay privacy concerns, it is important to note that all usage data is collected anonymously, and data can only be viewed at the population level. No identifying information will ever be collected without the express permission of the user.A rubric developed by Harry Walker at Johns Hopkins University [33] objectively measures the educational quality of iPod apps based on 6 domains (Table 3). While this rubric was initially developed for apps targeted at grade-school students, we plan to make minor modifications that will make it relevant to apps geared at medical professionals. Using the modified rubric, Canadian medical educators (including several from uOttawa’s Academy for Innovation in Medical Education) will be asked to objectively grade LearnENT. The modified rubric will be made publically available to all app developers.Table 3
Evaluation rubric for iPod Apps by Harry Walker. Reproduced with permission.
Domain1234Curriculum connectionSkill(s) reinforced in the app are not clearly connected to the targeted skill or conceptSkill(s) reinforced are prerequisite or foundation skills for the targeted skill or conceptSkill(s) reinforced are related to the targeted skill or conceptSkill(s) reinforced are strongly connected to the targeted skill or conceptAuthenticitySkills are practiced in a rote or isolated fashion (e.g., flashcards)Skills are practiced in a contrived game/simulation formatSome aspects of the app are presented an authentic learning environmentTargeted skills are practiced in an authentic format/problem-based learning environmentFeedbackFeedback is limited to correctness of student responsesFeedback is limited to correctness of student responses and may allow for student to try againFeedback is specific and results in improved student performance (may include tutorial aids)Feedback is specific and results in improved student performance; Data is available electronically to student and teacherDifferentiationApp offers no flexibility (settings cannot be altered)App offers limited flexibility (e.g., few levels such as easy, medium, hard)App offers more than one degree of flexibility to adjust settings to meet student needsApp offers complete flexibility to alter settings to meet student needsUser friendlinessStudents need constant teacher supervision in order to use the appStudents need to have the teacher review how to the use the app on more than one occasionStudents need to have the teacher review how to the use the appStudents can launch and navigate within the app independentlyStudent motivationStudents avoid the use of the app or complain when the app is assigned by the teacherStudents view the app as “more schoolwork” and may be off-task when directed by the teacher to use the appStudents will use the app as directed by the teacherStudents are highly motivated to use the app and select it as their first choice from a selection of related choices of apps


### 4.2. Subjective Assessment

Students’ insights into the roles of apps in their educational progress and outcomes will allow us, as well as future developers, to continue to create valuable educational tools in the future. This insight will be gathered through LearnENT’s in-app evaluation form, as well as through organized surveys of Canadian medical students, press reviews, and App Store ratings. Finally, the app will undergo peer-review by external experts, including members of the Canadian Society of Otolaryngology’sUndergraduate Medical Education Working Group.


### 4.3. Content Creation

Educational content should be tailored to the “learners’ perceived needs” and “address unrecognized deficits” [34]. When original content was determined, the focus was primarily on the learners’ needs and deficits that were identified by members of the OTOHNS faculty at the University of Ottawa, as well as the primary clerkship objectives listed in Table 1. The objective and subjective data gathered in the assessment phase will allow us to more accurately gauge the needs of the users and develop future materials accordingly.


### 4.4. Continued Innovation

As previously discussed, an appropriate level of challenge is a crucial component of learning modules. Future directions could include a novel form of case-based module that dynamically adjusts its difficulty level as a function of the user’s real-time performance. Additionally, future cases could support multiple outcomes (i.e., the learner’s decisions affect the outcome) as these have been found to promote the ability to adapt to a wide variety of situations [35], a trait that is highly valued in medicine.


### 4.5. Increased Accessibility

To ensure equality for all medical students, it is our goal to make LearnENT’s content available online. The content could be incorporated into an ecosystem that would centralize OTOHNS learning resources and promote collaboration between OTOHNS educators across Canada. As with LearnENT for iOS, all content shared within the LearnENT ecosystem would be freely available to all medical students internationally.


## 4.1. Objective Assessment

Although LearnENT has already undergone significant beta testing, the next phase involves more extensive review. Evaluating how, when, where and why students use the app will be paramount as we attempt to elucidate the role of future apps in the educational realm [32].Using existing technologies such as TestFlight and Flurry Analytics, we are able to develop a valuable understanding of how users interact with LearnENT. First, we will determine adoption and retention rates using statistics such as number of new and returning users, as well as session length and frequency of use. As it is free and easily available, we expect LearnENT to have a high adoption rate. We will be more interested; however, in the retention rate, as well as session frequency and length which will allow us to infer the users’ opinions of the app’s value. Additionally, we will track each section individually, allowing us to determine which sections are the most and least popular. This data will potentially allow us to extrapolate the preferred learning styles of our users, which will be used in the development of future material. Finally, user demographics such as country of origin can be used to create targeted content, including multilingual versions of the app. To allay privacy concerns, it is important to note that all usage data is collected anonymously, and data can only be viewed at the population level. No identifying information will ever be collected without the express permission of the user.A rubric developed by Harry Walker at Johns Hopkins University [33] objectively measures the educational quality of iPod apps based on 6 domains (Table 3). While this rubric was initially developed for apps targeted at grade-school students, we plan to make minor modifications that will make it relevant to apps geared at medical professionals. Using the modified rubric, Canadian medical educators (including several from uOttawa’s Academy for Innovation in Medical Education) will be asked to objectively grade LearnENT. The modified rubric will be made publically available to all app developers.Table 3
Evaluation rubric for iPod Apps by Harry Walker. Reproduced with permission.
Domain1234Curriculum connectionSkill(s) reinforced in the app are not clearly connected to the targeted skill or conceptSkill(s) reinforced are prerequisite or foundation skills for the targeted skill or conceptSkill(s) reinforced are related to the targeted skill or conceptSkill(s) reinforced are strongly connected to the targeted skill or conceptAuthenticitySkills are practiced in a rote or isolated fashion (e.g., flashcards)Skills are practiced in a contrived game/simulation formatSome aspects of the app are presented an authentic learning environmentTargeted skills are practiced in an authentic format/problem-based learning environmentFeedbackFeedback is limited to correctness of student responsesFeedback is limited to correctness of student responses and may allow for student to try againFeedback is specific and results in improved student performance (may include tutorial aids)Feedback is specific and results in improved student performance; Data is available electronically to student and teacherDifferentiationApp offers no flexibility (settings cannot be altered)App offers limited flexibility (e.g., few levels such as easy, medium, hard)App offers more than one degree of flexibility to adjust settings to meet student needsApp offers complete flexibility to alter settings to meet student needsUser friendlinessStudents need constant teacher supervision in order to use the appStudents need to have the teacher review how to the use the app on more than one occasionStudents need to have the teacher review how to the use the appStudents can launch and navigate within the app independentlyStudent motivationStudents avoid the use of the app or complain when the app is assigned by the teacherStudents view the app as “more schoolwork” and may be off-task when directed by the teacher to use the appStudents will use the app as directed by the teacherStudents are highly motivated to use the app and select it as their first choice from a selection of related choices of apps


## 4.2. Subjective Assessment

Students’ insights into the roles of apps in their educational progress and outcomes will allow us, as well as future developers, to continue to create valuable educational tools in the future. This insight will be gathered through LearnENT’s in-app evaluation form, as well as through organized surveys of Canadian medical students, press reviews, and App Store ratings. Finally, the app will undergo peer-review by external experts, including members of the Canadian Society of Otolaryngology’sUndergraduate Medical Education Working Group.


## 4.3. Content Creation

Educational content should be tailored to the “learners’ perceived needs” and “address unrecognized deficits” [34]. When original content was determined, the focus was primarily on the learners’ needs and deficits that were identified by members of the OTOHNS faculty at the University of Ottawa, as well as the primary clerkship objectives listed in Table 1. The objective and subjective data gathered in the assessment phase will allow us to more accurately gauge the needs of the users and develop future materials accordingly.


## 4.4. Continued Innovation

As previously discussed, an appropriate level of challenge is a crucial component of learning modules. Future directions could include a novel form of case-based module that dynamically adjusts its difficulty level as a function of the user’s real-time performance. Additionally, future cases could support multiple outcomes (i.e., the learner’s decisions affect the outcome) as these have been found to promote the ability to adapt to a wide variety of situations [35], a trait that is highly valued in medicine.


## 4.5. Increased Accessibility

To ensure equality for all medical students, it is our goal to make LearnENT’s content available online. The content could be incorporated into an ecosystem that would centralize OTOHNS learning resources and promote collaboration between OTOHNS educators across Canada. As with LearnENT for iOS, all content shared within the LearnENT ecosystem would be freely available to all medical students internationally.


## 5. Conclusion

When combined with good instructional design, eLearning can provide a learner the opportunity to engage in constructivist learning. LearnENT is an iOS app developed for medical students in OTOHNS. Its development was based on peer-reviewed pedagogical theories such as Laurillard’s design process, Honey & Mumford’s learning styles and Nielsen’s theory of learnability. To promote user friendliness, LearnENT offers a unique but intuitive user interface, simple tutorials, and timely help dialogs. To adequately engage users of all learning types, the app features content ranging from the traditional (simples text-based learning modules and links to peer-reviewed references) to the cutting edge (instructional movies, as well as interactive quizzes and modules that offer timely feedback). Potential user benefits include increased portability, fast and convenient access to reliable information, and improved retention of knowledge. While reviews from early beta testers have been overwhelmingly positive, plans are in place to perform further subjective and objective assessment. This will allow us to analyze the implementation phase and perform a quantitative and qualitative review on how learners utilize LearnENT. This data will help us, as well as other developers to continue to create relevant, accessible, and engaging apps for medical students, residents, and professionals alike.



---

*Source: 671383-2012-05-15.xml*