Inclusive Learning through Real time Tracking Display of Captions

Deaf and Hard of Hearing (DHH) students cannot follow classroom lectures without accommodations such as real time speech to text transcription. Current classroom transcription systems, such as C Print improve access to classroom lectures, but still do not provide equivalent access to spoken information. These transcription systems require the DHH students to watch the transcription on a personal laptop screen, which is suitable for speeches, but not engineering lectures. Unlike speeches, most engineering lectures include use of detailed visuals such as slides or diagrams, and sequential procedures. DHH students constantly look away from their laptop display to search and study the visuals. As a result, they spend less time watching lecture visuals and gain less information than their hearing peers. However, the need to process simultaneous aural and visual information can also be taxing for hearing students, and previous studies have shown that they also benefit from real time speech to text transcription. We evaluated the real time display of captions (RTD) usability by both deaf and hearing students in an engineering class. It further examined the factors that influence hearing students' use of RTD as an alternative source of information to help with their learning process in the classroom, and the factors that influence deaf students' use of RTD. Our evaluation showed that DHH students prefer a continuously moving RTD with three lines, and that is as close as possible to the teacher. On the other hand, hearing students prefer a RTD that has 6 lines that at a fixed location. Challenges Historically, DHH students are an under represented and under served minority in higher education because they do not receive adequate information in class. As a result these students are often unprepared for traditional STEM classrooms. Most DHH students cannot understand spoken lectures without the aid of aural to visual access. Prior to the introduction of the earliest accessibility laws in 1974, less than 4% of DHH individuals completed college in the 1960s. Although substantial progress has been made in the past 40 50 years in terms of accessibility laws and technology, the disparity in graduation persists: the graduation rate is 16% of DHH as compared to 30% for their hearing peers. Currently there are over 31,000 DHH students enrolled in college and this enrollment number is up 15,000 over the past 10 years. Deaf and hard of hearing students, and students with visual learning preferences are underrepresented in engineering, in part because it is difficult for visual learners to sustain attention on more than one visual simultaneously.Their attention is severely taxed when they have to switch attention between detail rich slide or demonstration visuals and the interpreter or teacher. The underlying reason is that the classrooms are not designed to utilize students' visual skills and are not fully accessible by DHH students, including engineering classrooms. When teachers maximize the benefits of visual learning, the barriers in regular lectures for DHH students, such as using spoken English is partially ameliorated.  The spread of visual information sources (interpreter, whiteboard, slides, teacher) in a classroom Hearing students are able to simultaneously watch the visuals and listen to the spoken explanation, while DHH students have to choose either the visuals or captions. Typically, they constantly look away from the real time transcript to search and observe details in the classroom visuals. As a result, they spend less time on the visuals and gain less than their hearing peers. They can also fall behind in reading the real time transcript compared with their hearing peers. Our paper compares viewing preferences between DHH and hearing students when they viewed two versions of real time transcript display interfaces with C Print, which had a 4 5 second delay. Our previous research has shown that DHH students preferred 3 lines, while hearing students preferred 6 lines. The feedback indicated that DHH students reported that captions was their only access to spoken information, so the 4 5 second delay between the speech and captions was not an issue. On the other hand, hearing students reported using the captions as backup, for example, to review information if they misheard or did not understand a specific word or sentence.  Real time text is usually delivered through a display system. One popular integrated captioning and display system is C Print, which was developed at the National Technical Institute for the Deaf, a federally funded institution for deaf and hard of hearing students. The system has significantly improved access to lectures for DHH individuals in many programs around the country. It also benefits individuals with other disabilities, such as those with a visual impairment or a learning disability. a (https:/www.rit.edu/ntid/cprint/ C Print displays the printed text of spoken English on a laptop/tablet in real time, which is a proven and appropriate means of acquiring information for some individuals who are DHH. A trained operator, called a C Print captionist, produces a text display of the spoken information in classroom or other settings. At the same time, one or more students read the display to access the information. The main advantage of the C Print system as compared with professional stenographers output is cost and availability. In terms of cost, C Print typists need far less training and time as compared with stenographers who and are cheaper than professional stenographers, who type on a specialized keyboard using short hand notation  Although stenographers are more accurate, studies have shown that C Print typists' accuracy is high enough that studies have not found any significant difference between C Print and sign language interpreters in terms of student learning in the classroom [19, 4] The main disadvantage of C Print, like CART, is that the "flow" of text is not smooth or consistent, unlike LegionScribe [10] People often falsely assume that the use of captions in the classroom enables full access to realtime text for DHH people. This assumption minimizes other information accessibility issues, such as simultaneous visual streams or content complexity. For example, DHH students have to constantly switch gaze between the C Print display and the lecture slides. So, for current classrooms, captions should be presented in a way that reduces the effects of visual dispersion and cognitive overload. Our RTD system addresses the challenges described in the previous section and in previous work. The goal of our RTD system is to minimize visual dispersion and maximize use of captions for the caption viewers. The system involves three components: C Print, Kinect 2, and the RTD system. The system can be easily set up and turned on within a few minutes by the captionist. For capturing and displaying the captions, the RTD system uses a laptop and application to integrate the C print application with a Kinect 2 device, so that the display of real time text by the C Print captionish is displayed over the head of the person. The RTD system is mounted on a cart so that it can be moved and quickly deployed. The RTD real time transcript is produced by C Print above the presenter, which is all within the viewer's peripheral vision as shown in Figure 2.  RTD System with captions The RTD system uses the Kinect 2 system to track and record the current location of the presenter, as shown in Figure 5. The system records the location of the presenter, and displays the projected captions at a fixed distance from the presenter's body, usually over their head. This projection scheme ensures that the captions and presenter are always close to each other at a fixed distance. The fixed distance and location predictability enables the viewer to easily see both the transcript and presenter's information simultaneously. They can detect changes, easily follow and read the real time transcript without missing the presenter's information. The startup time is less than a minute and mostly the time for the projector to start up, which is about 30 seconds. The cost of the entire system currently ranges from about $700 to $1000. The bulk of the cost is to cover a laptop, such as a Surface Pro for $500. The other items are somewhat less expensive, ranging from $200 for the Kinect 2 for Windows system at the low end. The final components of the system include the provision of a cart to provide easy transportation of the system from one place to another and vice versa.  We ran a study in a class that was about 2 hours long for each lecture. We asked the students to evaluate the readability and usability of the RTD captions, with varying amount of lines and movement. Prior to the experiments, we displayed RTD in a static location in two class lectures to let students become comfortable with the technology. The study was conducted over four lectures. On the first day, the captions were displayed at fixed location with three lines (STD3) On the second day, the captions were displayed above the teacher with three lines (RTD3) On the third day, the captions were displayed at a fixed location with six lines (STD6) On the fourth day, the captions were displayed above the teacher with six lines (RTD6) At the start of each class, we announced that we were evaluating speech to text accommodations, and that we would distribute a survey about the speech to text display at the end of class, and that it would take about a minute to complete. We distributed the surveys and collected them at the end of class, after they were completed.Our survey consisted of a demographic question on hearing status, four Likert questions on user preferences, and two open ended questions to solicit feedback on their experience with the system in action. All of the enrolled students in attendance for th