Computing in Japanese
what are the frontiers now?
Jim Breen
Monash University(jim.breen@infotech.monash.edu.au)
Introduction
A workshop on Computational Japanese Studies provides an opportunity to take stock of what
exactly is computing in Japanese, and how it differs, if at all, from language­processing activities inother languages. Where such differences exist, is is appropriate to consider whether they are still
relevant today, when there has been a massive investment in such things as internationalized “singlebinary” software and unified character sets. Attention is also needed as to where computertechnology can have useful impact in Japanese studies, and identifying where priorities should beplaced.
Computing in Japanese
Why do we even talk about computing in Japanese or Computational Japanese Studies? We do nottalk about Computational Dutch Studies, or Computational Italian Studies. In this author’s view, thestandout reason for this is the Japanese orthographical system. The mixed kanji/kana system,
combined with other aspects of the orthography, leads to a number of issues which played asignificant part in the introduction of information technology in Japan. Among these issues are:a. encoding of kanji and kana in files
b. representation of text (display, print, etc.)
c. input of text (by humans)d. segmentation of text into lexemes
e. canonicalization of accepted variants
All of these, and in particular the first three, had a major impact on the adoption of IT in Japan, andare considered by many to have led to a slower uptake of IT than in other countries with equivalentlevels of industrialization. The importance of the issues listed above can be seen from the time it
took to address them comprehensively. (As a test, consider the situation if Japan had adopted atotally romanized writing system in the early 20th century, as did nations such as Turkey andMalaysia. None of the above would have been considered issues in the introduction and use ofcomputing.)
Encoding: It took over two decades from the introduction of computing in Japan before a nationalstandard for kana and kanji coding was established, and many years elapsed before it was widelyadopted.
Representation: Having two orders magnitude more characters to deal with certainly strained thecapacities of end­user facilities. Interim low­storage and complexity solutions such as hankaku
katakana, were used for many years (and persist today.) Solutions only arrived with technologicaldevelopments such as VLSI, low­cost storage and non­impact printing.

Text Input: This was a major issue for many years, with complex single­kanji selection systems
persisting until the late 1980s and beyond. Unger in his 1987 book “The Fifth Generation
Fallacy”[3] asserted that the main goal of that project, launched in 1981, was to overcome theproblem of Japanese text input through heavy use of AI techniques.
Current Situation
All of the issues listed above were comprehensively addressed at the technological level during the1980s and 1990s, and with the impact of internationalization many of the solutions have been
embedded as standard elements in software. For example, virtually all major operating systems
releases now have as installation options full support for Japanese input, display and printing (alongwith support for many other languages and scripts). Most high­level languages support non­alphanumeric text handling. Japanese text segmentation, which in the 1980s was regarded by manyas an intractable problem, can now be performed effectively by several open­source and commercialsystems.
Thus, the majority of orthography­related issues which tended to dominate the early stages ofcomputing in Japanese have been adequately resolved. Computing in Japanese can validly be seen
as on the same footing as computing in languages using alphabets, and the focus of “computationalJapanese” is now largely on issues related to the language itself.
The Frontiers
While the application of computer technology to Japanese studies is now in a similar position toother languages and cultures, and in areas such as NLP faces the same challenges, there are several
topics which are worthy of particular attention. Addressing these topics should be made a priority inthe application of computer technology. The following is proposed as a short­list of “frontier” topic
which could well do with attention:
a. Dictionaries. Sue Atkins noted over a decade ago that computerization seemed to have
limited impact on the user aspects of dictionaries, even if they were available on CDROM,
and that “underneath these superficial modernizations lurks the same old dictionary”[2].
Japanese probably has the highest density of dictionaries of any language, and certainly large
numbers are available electronically, but despite, or perhaps because of standards likeEPWING/JIS X 4081, access to and presentation of dictionary content is still largely areplication of paper dictionary techniques. Atkins proposed a number of areas where
computational resources should be exploited in a “new­age” dictionary, including extensive
user customization, use of hypertext, etc. There is certainly scope for study in this area. In
addition, there is a paucity of lexicons that are readily and freely available for research.
Effort should be put into extending the free lexicons that are available, or seeking the freeing
up of sources such as the EDR collection of lexicons, which are currently too expensive formany researchers.
b. Corpora. Japanese is not particularly well served in the area of available corpora, and is
especially poor in the area of parallel bilingual and multi­lingual texts. While modest
numbers of bitexts can be identified, they tend to be under commercial restrictions, and aregenerally unavailable for wide exploitation. The establishment of a comprehensive andrepresentative Japanese corpus, and in particular the assembly of accurate bitexts, should bea priority.

c. Computer­Assisted Language Learning (CALL). For some reason CALL seems to be a“difficult” area for research. There are myriads of systems around, many of themcommercial, yet few seem to get past being yet­another flashcard or vocabulary drilling tool.
There has been little real research into the efficacy of such systems. One suspects that theproblem lies in the gap between language education specialists and designers/developers ofsoftware. Given the popularity of Japanese study worldwide, (and indeed the popularity ofEnglish study in Japan), there is surely scope for proper research into where CALL has thegreatest potential and which types of CALL tools are most effective.
d. Text Searching. With WWW search engines playing an important part in modern life, it isimportant that Japanese text is handled appropriately. Leading search companies such as
Google and Yahoo apply a common framework for all languages, which at times does notcompletely cope with aspects of Japanese orthography, such as multiple written forms ofwords[2]. There is ample scope for more work in this area.
e. Machine Translation. In many ways MT into or out of Japanese is in no different a situation
as other languages. There is a small number of reasonable, but expensive commercialsystems (e.g. Fujitsu’s ATLAS), and a large number of inexpensive but poorly performing
systems. Most R&D work seems to take place in commercial organizations, and hence IP
issues preclude significant sharing of lexicons, etc. or even significant publication ofmethodologies. The area of statistical MT, which perhaps is seeing more non­commercial
activity than traditional techniques, is hampered by the limitations in the availability ofJapanese­Other parallel texts and freely available lexicons. This area of MT would greatly
benefit by the expansion of readily available corpora and lexicons, as suggested above.
References
1. BTS Atkins, Bilingual Dictionaries: Past, Present and Future, Euralex'96, reprinted in
Lexicography and Natural Language Processing: A Festschrift in Honour of B.T.S. Atkins, Euralex,
2002.
2. JW Breen, WWW Search Engines and Japanese Text, Sixth Symposium on Natural Language
Processing 2005 (SNLP 2005), Chiang Rai, Thailand, December 2005
3. JM Unger, The Fifth Generation Fallacy: Why Japan is Betting Its Future on ArtificialIntelligence, Oxford University Press, 1987