Development of Multimodal Textbookswith Invisible 2-Dimensional Codes
for Students with Print Disabilities
Akio Fujiyoshi1 , Mamoru Fujiyoshi2 , Akiko Ohsawa2 , and Yuko Ota3
1

Ibaraki University, Hitachi, Ibaraki
fujiyosi@mx.ibaraki.ac.jp
2
National Center for University Entrance Examinations, Meguro, Tokyo{fujiyosi,ohsawa}@rd.dnc.ac.jp
3Dai-ni Enzan Elementary School, Shinagawa, Tokyoota-y@city.shinagawa.tokyo.jp

Abstract. Utilizing invisible 2-dimensional codes and digital audio players with a 2-dimensional code scanner, we developed a new type oftextbooks for students with print disabilities, called “multimodal textbooks.” Multimodal textbooks can be read with the combination of thetwo modes: “reading printed text” and “listening to the speech of thetext from a digital audio player with a 2-dimensional code scanner.”
Since a multimodal textbook looks the same as a regular textbook andthe price of a digital audio player is reasonable (about 30 euro), we thinkmultimodal textbooks are suitable for students with print disabilities inordinary classrooms.

1Introduction

In 2012, Japanese Ministry of Education, Culture, Sports, Science and Technology published a report about students with developmental disorder in ordinaryclassrooms in compulsory education. According to the report, 2.4% of students
in ordinary classrooms have remarkable diﬃculties in reading or writing without
intellectual retardation. This means that most students with print disabilities
are in ordinary classrooms and they use regular textbooks.
Utilizing invisible 2-dimensional codes and digital audio players with a 2dimensional code scanner, we developed a new type of textbooks for students
with print disabilities (Fig. 1). We can read these new textbooks with the combination of the two modes: “reading printed text” and “listening to the speechof the text from a digital audio player with a 2-dimensional code scanner.” So
we call them “multimodal textbooks.” Since sight and hearing help each other,
we can read them easily and correctly. Authors have developed auditory testing
media for test-takers with print disabilities [1, 2, 3]. To create multimodal textbooks, we took advantage of the experience and technique gained during thedevelopment of auditory testing media.
For students with print disabilities, large print textbooks and DAISY textbooks have been available in Japan. Recently, textbook publishing companies
K. Miesenberger et al. (Eds.): ICCHP 2014, Part II, LNCS 8548, pp. 331–337, 2014.
c Springer International Publishing Switzerland 2014
332

A. Fujiyoshi et al.

Fig. 1. Multimodal Textbooks

started producing large print textbooks by themselves at their expense. Thequality of large print textbooks is very improved. DAISY (Digital Audio Accessible Information System) is a world standard of accessible books for people withprint disabilities [4]. A number of textbooks are translated into DAISY by theeﬀorts of volunteers. Many students with print disabilities in special supportedclassrooms use either large print textbooks or DAISY textbooks. However, thenumber of students in ordinary classrooms who use these textbooks is very few.
A multimodal textbook consists only of a paper textbook and a digital audio
player with a 2-dimensional code scanner. Since the paper textbook looks thesame as a regular textbook and the digital audio player is small and can beused with a headphone, multimodal textbooks can be easily used in ordinaryclassrooms. Multimodal textbooks have the following other features:– They are easy to carry.
– The price of a digital audio player is reasonable (about 30 euro).
– Students can write memo on the paper textbooks with a pencil or a ﬂuorescent marker.
– Students can study by themselves.
– Teachers can easily get to know positions where students are reading.
In 2011, the ﬁrst trial version of multimodal textbooks was made. With the trialversion, multimodal textbooks were demonstrated to teachers, researchers andeditors of textbooks. From October 2012, a practical evaluation started with 8
elementary school students with print disabilities in Shinagawa, Tokyo. FromApril 2013, the practical evaluation was continued with 30 elementary school
students in Shinagawa, Tokyo.
A semiautomatic production system for multimodal textbooks was developedin order to ease the production process of multimodal textbooks. From a PDF ﬁleof a textbook, the semiautomatic production system automatically recognize thelayout of the textbook, arrange 2-dimensional code on each page of the textbook

Development of Multimodal Textbooks with Invisible 2-Dimensional Codes

333and produce a manuscript of the speech of the text. The system also has aneditor for corrections.
Currently, all 10 titles of elementary school textbooks in Japanese language
published by Mitsumura Tosho Publishing Co.,Ltd., all 3 titles of junior highschool textbooks in Japanese language published by Mitsumura Tosho Publishing Co.,Ltd., all 12 titles of elementary school textbooks in Japanese language
published by GAKKO TOSHO Co.,Ltd, and 2 titles of junior high school textbooks in English language published by SANSEIDO Co.,Ltd. are available asmultimodal textbooks.

2Development of Multimodal Textbook

2.1Invisible 2-Dimensional Codes and a Digital Audio Player

The introduction of invisible 2-dimensional codes and digital audio players witha 2-dimensional code scanner enable us to develop multimodal textbooks.
We employ ‘screen code’, an invisible 2-dimensional code system developed by
Apollo Japan Co., Ltd. Screen code consists of dots in 2 mm square (Fig. 2). Since
the intervals are large enough for the size of dots themselves, dots are almostinvisible. If we use invisible ink, which absorbs only infrared light, instead of
black ink, dots become totally invisible.

Fig. 2. Dots of ‘screen code’

As a reading device for multimodal textbooks, we employ ‘Speakun’ developedby Apollo Japan Co., Ltd (Fig. 3). Speakun has a 2-dimensional code scanner at
its top. When a 2-dimensional code is scanned with Speakun, the corresponding
speech sound is reproduced. We can listen to the sound through a headphone orbuilt-in speaker. The sound volume and speed can be adjusted with its buttons
mounted at the front side. The sound data is stored in a micro SD card. 2G byte
is enough to store all sound data of all textbooks of a student.

334

A. Fujiyoshi et al.

Fig. 3. Speakun

2.2

Semiautomatic Production System for Multimodal Textbooks

Currently, multimodal textbooks are produced by a small number of staﬀs. In
order to ease the production process of multimodal textbooks, a semiautomatic
production system for multimodal textbooks was developed (Fig. 4). The systemis written in Java, and takes advantage of software packages Ghostscript [5] andPDFMiner [6]. From a PDF ﬁle of a textbook, the semiautomatic production
system automatically recognize the layout of the textbook, arrange 2-dimensional
code on each page of the textbook and produce a manuscript of speech sound.
The system also has an editor for corrections.

Fig. 4. Semiautomatic production system for multimodal textbooks

Development of Multimodal Textbooks with Invisible 2-Dimensional Codes

335

The assignment of speech sound ﬁles to 2-dimensional codes can also be donewith the system. A sound ﬁle can be assigned to a set of rectangular areas toprint the same 2-dimensional code by a drag-and-drop operation.
With this system, arranging 2-dimensional code on a textbook of 300 pages
can be done in 20 hours by a person, which is 5 times faster than without thesystem. 27 titles of multimodal textbooks have been made so far.

3

Design of Multimodal Textbooks

Multimodal textbooks were designed so that they help to improve users’ abilityof reading. Thus the “reading printed text” mode should take precedence overthe “listening to the speech of the text” mode. In other words, while users arelistening to speech sound, we want them to read the corresponding printed text
by their sight. Therefore, we put a diﬀerent 2-dimensional code to each sentences
and cut speech sound ﬁles into small pieces so that only corresponding speechsound is reproduced by one scan.
Fig. 5 shows a page of a multimodal textbook in English language for juniorhigh schools (New Crown 1, SANSEIDO Co.,Ltd.). Every rectangle is assigned
to a diﬀerent 2-dimensional code except rectangles connected by brown lines. A
set of rectangles connected by brown lines is assigned to the same 2-dimensional
code.
Users have to scan a 2-dimensional code every time they want to read the nextsentence. In order to scan the 2-dimensional code of the next sentence smoothly,users have to trace printed characters of the current sentence.

4

Results of a Preliminary Investigation on Multimodal
Textbooks

In August 2013, we made a preliminary investigation in order to check how
multimodal textbooks had been used in class and at home. We delivered questionnaires to the users of multimodal textbooks in a practical evaluation inShinagawa, Tokyo. Since the most users are elementary school students, thequestionnaires were entered by a parent or a teacher of them. We have received
15 answered questionnaires from the users of multimodal textbooks. They werefrom 8 elementary school students in lower grade, 6 elementary school students
in higher grades, and 1 junior high school student. The percentage of reply forthe questionnaires was 50%. The results are as follows:
4.1

Usage of Multimodal Textbooks

Among the 15 users, 13 persons (87%) mostly use multimodal textbooks at home
for preparation and review. Only 2 persons regularly use multimodal textbooks
in class.

336

A. Fujiyoshi et al.

Fig. 5. A page of a multimodal textbook in English language for junior high schools(New Crown 1, SANSEIDO Co.,Ltd.)

4.2

Impressions from Students

The following impressions were received from students: “It is convenient to readbecause Chinese characters (kanji) are read by the machine.” “It is easy to study
intonation and punctuation of a sentence.” “I can freely choose a sentence toread.” “Because of sound, a lot of sentences can be read.” “The speech speed
control is convenient.” and “It is good because we can write the reading of aChinese character in kana at its side.”
The following requests were also received from students: “Please make anyother books be like multimodal textbooks.” and “I want to take tests withsound.” We think they are our big future works.

Development of Multimodal Textbooks with Invisible 2-Dimensional Codes

4.3

337

Impressions from Parents and Teachers

The following impressions were received from parents and teachers: “A multimodal textbook is easy to use. My child can use it by himself.” “My child canstudy at her own pace with a multimodal textbook.” “Children are very interested in mechanism of digital audio players with a 2-dimensional code scanner.”and “A multimodal textbook is a pleasant tool since students can use it bythemselves for preparation and review.”

5Conclusion

Utilizing invisible 2-dimensional codes and digital audio players with a2-dimensional code scanner, we developed a new type of textbooks for students
with print disabilities, called “multimodal textbooks.” Since we can read multimodal textbooks with the combination of “reading printed text” and “listening
correspondent speech sound to the text”, we can read them easily and correctly.
We want to study other designs of multimodal textbooks. Since the current
version of multimodal textbooks was designed so that they help to improve users’ability of reading, it might be suitable for students with print disabilities whose
disorder is relatively mild. We plans to make multimodal textbooks with document structure diagrams for students with print disabilities whose disorder isrelatively severe. The usage of document structure diagrams has been studied
in the development of auditory testing media for test-takers with print disabilities [3].
There are 27 titles of multimodal textbooks available for students with print
disabilities. From April 2014, we plan to deliver multimodal textbooks nationwide in Japan.

References
1. Fujiyoshi, M., Fujiyoshi, A.: A new audio testing system for the newly blind and thelearning disabled to take the National Center Test for University Admissions. In:Miesenberger, K., Klaus, J., Zagler, W., Karshmer, A. (eds.) ICCHP 2006. LNCS,
vol. 4061, pp. 801–808. Springer, Heidelberg (2006)
2. Fujiyoshi, M., Fujiyoshi, A., Aomatsu, T.: New Testing Method for the Dyslexic and
the Newly Blind with a Digital Audio Player and Document Structure Diagrams.
In: Miesenberger, K., Klaus, J., Zagler, W., Karshmer, A. (eds.) ICCHP 2010, Part
1. LNCS, vol. 6179, pp. 116–123. Springer, Heidelberg (2010)
3. Fujiyoshi, M., Fujiyoshi, A., Ohsawa, A., Aomatsu, T., Sawazaki, H.: Developmentof New Auditory Testing Media with Invisible 2-Dimensional Codes for Test-Takers
with Print Disabilities. In: Miesenberger, K., Karshmer, A., Penaz, P., Zagler, W.
(eds.) ICCHP 2012, Part I. LNCS, vol. 7382, pp. 116–123. Springer, Heidelberg(2012)
4. DAISY, http://www.daisy.org/
5. Ghostscript, http://www.ghostscript.com/
6. PDFMiner, http://www.unixuser.org/~ {}euske/python/pdfminer/