日本ソフトウェア科学会第 39 回大会 (2022 年度) 講演論文集

An Ongoing Design of Dictating Programming
That Accepts Noisy Input
Wennong Cai, Shigeru Chiba

We propose a “predict before parsing” architecture to let the dictating programming system be able tohandle noisy input. We think the user experience of existing dictating programming designs is restricted by
the quality of speech recognition systems, and the carefulness of users speaking. The eventual goal of theproposed architecture is to parse a dictating programming statement into its Abstract Syntax Tree, even if
the statement contains noises.
The current progress of building this architecture and several novel points in the implementation areintroduced in this paper. We conducted experiments on the “Statement Predictor”, which is a component
part of the architecture. The results show that an appropriately trained machine learning model can predict
the type of noisy statement with high accuracy.

ﬁciency of doing programming on portable devices.

1 Introduction

In this paper, we ﬁrst introduce the background

Programming is essential nowadays, and thereof dictating programming and the issue of noises.

are lots of engineers making their products by typ-Then we present our proposal of a ﬂexible parser

ing the programming code on the keyboard. How-

with the statement predictor as the solution toever, as electrical devices become more and more

the issue. Ongoing experiment results are shownportable, the way to do the programming still re-after that.

quires a large keyboard to achieve the best eﬃ-

lated works on dictating programming, and com-ciency. Portable devices such as tablets or smart-pare their works with this paper.

Finally, we also introduce some re-phones are seldom used to type programming, as
the tiny-size touch-screen keyboard is hard to typepunctuation symbols or indents.

2 Background of Dictating Programming

Dictating Programming is an approach to im-

The idea of dictating programming is to use

prove feasibility of programming on small devices.

speaking instead of typing to enter the program-

As it is common nowadays that a phone is equippedming code. A dictating programming system gen-with a speech recognition system, being able to use

erally has two sub-system parts. The ﬁrst part isthe voice to support the programming improves ef-the speech recognition part, which is responsible∗ ノイズ入力を受け入れる口述プログラミングの設計

This is an unrefereed paper. Copyrights belong tothe Author(s).
蔡 文農, 千葉 滋, 東京大学大学院情報理工学系研究科,
Graduate School of Information Science and Technology, The University of Tokyo.

for converting human speaking voice into text. The
second part is an algorithm that translates the recognized spoken text into an existing programming
language, or executes it directly. However, one uncertain point in such a system is that the recognizedspoken text is hardly as perfect as the algorithm input, because it often contains noises. We consider
a dictating programming should allow a certain extent of noises.

3 Proposal:

Flexible

ParserwithStatement Predictor
To address the issue of parsing noisy dictating

Human Speaking and Speech Recognition are two

programming, we propose an architecture that con-primary noise sources considered in this research.

tains a ﬂexible parser with statement predictor, as

We came to this decision by reading some sim-shown in Figure 1. Details of overview design andple Python code to the existing speech recogni-

the statement predictor are introduced in this sec-tion system, in a format of dictating programmingtion, while the parser component is still ongoinggrammar we designed. The recognition results areresearch.

collected and compared with the intended spoken
statement, and the diﬀerences between them areconcluded as the following two categories.

3. 1 “Predict before Parsing” Architecture
We made a hypothesis that for a noisy spoken

2. 1 Noises from Human Speaking

statement, knowing its intended statement type in

Speaking a sentence strictly following the valid

advance improves the accuracy of parsing it into its

grammar is exhausting, especially for foreign peo-intended version. To examine this idea, the “Pre-ple. People tend to speak a sentence in the mostdict before Parsing” architecture is proposed.

natural way they feel. Therefore, it is reasonable

The “Predict before Parsing” architecture isto assume dictating programming users can hardly

made of a Statement Predictor and a customized

speak the absolute correct grammar for the mostparser. The statement predictor is a trained ma-input. Some of the most common examples arechine learning model that predicts the statement

adding additional article words such as “the” ortype given the noisy input statement. The cus-“a”, using diﬀerent tenses for verbs, and using sin-tomized parser behaves like a normal parser whengular or plural forms for nouns.

parsing a valid statement, but also attempts toparse the noisy statement by taking guess action

2. 2 Noises from Speech Recognition

based on the predicted type, instead of just throw-

The existing Speech Recognition systems are noting the exception.

The system takes one input

good enough. They have the possibility of mak-statement at a time, and outputs the Abstract Syn-ing mistakes on recognizing words. Learning fromtax Tree for this statement.

large natural language datasets is also a key to highrecognition accuracy for most speech recognition
systems. However, such highly accurate recogni-

3. 2 Input Pre-Processing
We propose input preprocessing steps for our pro-tion does not apply to this research, since there isposed architecture.

no large available corpus of dictating programmingcreases the accuracy of the Statement Predictor,speaking.

and also improves dictating programming users experience.

Proper input processing in-Figure 1

The “Predict before Parsing” Architecture

3. 2. 1 Programming Language for Dictating

directly using character-based encoding, we turnevery token into its phonetic-based format ﬁrst,

A domain-speciﬁc language for dictating pro-then map each phonetic character to a numericgramming is designed in this research. Traditional

value. The Oxford Dictionary API [1] is used totyping programming language is hard to dictate,

retrieve the phonetic format automatically.

because there are many punctuation symbols in

3. 2. 3 Flatten Grammar Structureuse. Therefore, we made our own language withWe decide to divide any statement type contain-grammar that does not use punctuation symbols.

ing nested structure into several independent types

As an example, the following Python code:in order to ﬂatten the nested structure. For in-

1
2
34stance, a normal FOR-LOOP statement usually

if x > y :print ( x )else :
print ( y )contains the keyword

sion, the nested body statements, and the optional
keywordcan be translated into the following equivalentdictating programming statement:
if x is greater than y then invoke print with x else
invoke print with y
3. 2. 2 Phonetic-based Encoding
A novel encoding approach based on phonetics is
proposed in this research. The traditional tokenbased encoding approach in natural language ﬁeld
is not suitable in the task of dealing noisy sentences,because it treats two similar tokens as completelyindependent. The normal character-based encoding captures the similarity if a token is mis-spelled,
but the most common noises in dictating programming are mis-pronunciation. Therefore, instead offor , the condition expres-else

with its nested body statements.

In our dataset, we treat the FOR-LOOP statement
as 3 separate parts: The FOR-LOOP header, whichcontains the keyword for as well as condition expression; The body statements, which are no longer
part of the FOR-LOOP type but parallel independent statements; The FOR-LOOP ender, which isjust a static value

end for .

Mismatching errors between header and ender
are then handled during the program execution
phase. The ﬂatten strategy makes the parser output not directly contain any nested statement such
as FOR-LOOP or IF. As the drawback of this strategy, syntax errors such as providing an ELSE state-ment without an IF-header statement, cannot be

ply the bidirectional LSTM model [6] to let thecaptured during the parsing phase. We consider

model focus on both ends of the sequential input.

that this is acceptable since they can still be detected when executing the program.

4. 2 Dataset Generation

Flattening nested statements is a key to reduc-The dataset is generated from existing program-ing the diﬃculty of Statement Predictor training.

ming corpus into the format of our speciﬁc pro-

The nested statement usually has a longer length

gramming language for dictating as mentioned inthan non-nested statement, and longer input makes

section 3. 2. 1.

the model harder to ﬁnd useful information. Also,structed from the raw python code, then converted

the existence of inner statements interferes with theinto dictating programming statements by travers-prediction on the outer statement.

ing from each statement-level node. Each converted

Rejecting all nested statements as input is also
not feasible. Nested statement examples such as

The abstract syntax tree is con-statement will be stored along with its statement
type.

the deﬁnition of function or class, or the if-else

The python corpus made by the BIFI team isstatement, they are widely used in most existingused in this research. BIFI [8] is research on the au-

programming languages.

Rejecting the usage oftomatic correction of programming syntax errors.

those statement types makes a programming lan-

They collect 3 million Python 3 snippets which in-guage not practical.

clude valid code and syntax error code. We consider
the syntax errors in typing programming, such as

4 Experiments

missing punctuation symbols, do not represent theWe built a Statement Predictor and conductedsituation in dictating programming. Therefore, weexperiments by training it in diﬀerent approaches.

only make use of the valid code dataset as the in-

The Statement Predictor is evaluated by the ac-put source of the dataset generation process in ourcuracy of predicting the statement type of aresearch.

potentially-noisy input statement.

There are 18 labels in the original nested gram-Also, a large-scale dataset is required to train

mar structure, and 24 labels in the ﬂattened gram-a Statement Predictor. Due to the lack of exist-

mar structure. To prevent extreme imbalance ining popularly used dictating programming systems,

the label distribution, each label can have at mostthere is no publicly available dataset of dictating

1 million entries.

programming corpus. The approach of making aThe nested structure dataset contains 6520568

dictating programming dataset based on existingstatements, and contains 19964626 statements af-programming corpus is introduced in this section.

ter the grammar has been ﬂattened. The reason
for the size increment in the ﬂattened dataset is4. 1 Model Usedbecause many statements type such as FOR LOOP

The Long Short-Term Memory Model [7] is usedstatement have been divided into multiple indepen-as the Statement Predictor model in this paper.

dent statement types.

The LSTM model is a practical model to deal withWe have also made a smaller version of the ﬂat-sequential input data, but we found the LSTM intened structure dataset by setting a smaller maxi-

our task tends to focus on the last few tokens more

mum cap on each label, since it contains much morethan the beginning tokens. Therefore, we also ap-data entries than nested dataset. The light version

ModelUsing Noisy Identiﬁers

Grammar Structure

Dataset size

Accuracy

LSTM

No

Nested6520568

97.2%

LSTM

Yes

Nested6520568

26.5%

Bi-LSTM

Yes

Nested652056833.9%

Bi-LSTM

Yes

Flattened

358753192.3%

Bi-LSTM

YesFlattened

19964626

97.6%

Table 1

Experiment Results.

of the ﬂattened structure dataset contains 3587531
statements.

As Table 1 shows, the accuracy is high when every identiﬁer is treated as the same, with only somekeywords been modiﬁed to similar pronunciation

4. 3 Noise Simulation

words. It shows that under a relatively low noises

Introducing noises to the training dataset is vitalenvironment, the LSTM can recognize the state-

to increase the generality of the Statement Predic-

ment type based on a sequence of phonetic charac-

tor. While there is no publicly available dictatingters input.

programming corpus, noises have to be generated
in an imitation approach as well.

However, when introducing the noises on identiﬁers by using the 9012 nouns mentioned in sec-

For reserved keywords, several pre-deﬁned noisytion 4. 3, the accuracy has a great fall. The model

tokens have been selected that have similar pronun-

almost cannot recognize any labels but just gives

ciations. The keywords are replaced by those noisy

random guesses on the most popular labels. Usingtokens based on a random probability.

a bidirectional LSTM slightly improves accuracy,

For identiﬁer names, the original identiﬁers used

but still remains at an impractical level.

in Python code are already random noises, but

We think the reason for low accuracy is that iden-

most of them are hard to dictate or impossible totiﬁers in programming do not make any contribu-

be converted into the phonetic format. We pre-tion to determining the statement type. Also, re-pare a dataset with no identiﬁer noises by replac-

served keywords in the nested inner statements doing all identiﬁers with the same token. To simulate

not help to predict the outer statement type as well.

noises from identiﬁers, we use a collection of 10000

Having too much noisy information in the input

random nouns [5], and pick 9012 of them whichmakes the model hard to learn useful information.

have phonetic values in Oxford Dictionary API [1],In the ﬂattened dataset, the number of identiﬁers

then replaced identiﬁer names with one of themdecreases since there are no more inner statements.

randomly to make another dataset. The “UsingAll keywords left in the input are useful to predict

Noisy Identiﬁers” ﬁeld in Table 1 indicates which

the statement type as well. Accuracy gets a large

dataset is used.

increment as the results show.

4. 4 Results

4. 5 Limitations

All experiments are conducted in 4 epochs.

We considered 2 possible drawbacks in the cur-

Dataset is shuﬄed and divided into training, vali-

rent design that can be further improved in future

dation, and testing dataset in the portion of 8:1:1.

research. The ﬁrst drawback is the mapping fromphonetic characters to numeric values is intuitively

the user’s burden. We aim to enable the dictat-

designed in Phonetic-based Encoding. The similar-

ing programming system to accept the spoken text

ity in numeric values does not accurately represent

even if it contains some noisy tokens.

the similarity in pronunciation. The second drawback is the grammar of dictating programming used

6 Conclusionin this research is simply designed. The primary

In this paper, we introduced the issue of noises

goal of designing this grammar is just to removethat keeps the dictating programming not feasible

the necessity of using punctuation symbols, so allenough. The architecture design of a ﬂexible parser

tokens used are normal words that do have phonetic

with the statement predictor is presented. As on-form. A more delicate design of the grammar may

going research, we presented the current progress of

reduce the diﬃculty of handling noises, and further

some promising experimental results for the State-improve the feasibility of dictating programming.

ment Predictor implementation. Limitations of theStatement Predictor, such as phonetic encoding

5 Related Works

and grammar design, have also been presented as

There are existing attempts at designing the dic-potential future research. For our next step, we will

tating programming system. VoiceGrip [3] designs

attempt to design the customized parser in our pro-a domain-speciﬁc language that is easy-to-dictate,

posed architecture, which parses noisy statements

and can be translated into the C language. It also

given the predicted statement type.

designs an algorithm to recognize consecutive tokens as one single camel case identiﬁer, which en-参 考 文 献hances the ability to name identiﬁers in dictating[1] Oxford Dictionaries API.
Retrieved from:https://developer.oxforddictionaries.com.
[2] Cai, W., Akiyama, S., and Chiba, S.: A Preliminary Design of an Easy-to-Dictate Programming
Language with Pronouns, Proceedings of the 38th
JSSST Annual Conference, 2021.
[3] Desilets, A.: VoiceGrip: a tool for programmingby-voice, International Journal of Speech Technology, Vol. 4, No. 2(2001), pp. 103–116.
[4] Desilets, A., Fox, D., and Norton, S.: Voicecode:
An innovative speech interface for programmingby-voice, In CHI’06 Extended Abstracts on Human Factors in Computing Systems, (2006 Apirl),
pp. 239–242.
[5] Eric Price, MIT: 10000 Word List.
Retrieved from:https://www.mit.edu/ ecprice/wordlist.10000.
[6] Graves, A., Fernández, S., and Schmidhuber, J.:
Framewise phoneme classiﬁcation with bidirectional
LSTM and other neural network architectures, Neural networks, Vol. 18, No. 5-6(2005), pp. 602–610.
[7] Hochreiter, S. and Schmidhuber, J.:Long
Short-Term Memory, Neural Computation, Vol. 9,No. 8(1997), pp. 1735–1780.
[8] Yasunaga, M. and Liang, P.: Break-it-ﬁx-it: Unsupervised learning for program repair, Proceedingsof the 38th International Conference on Machine
Learning, PMLR, Vol. 139, 2021 July, pp. 11941–11952.

programming. In 2021, our previous dictating programming design, Natural Programming Language[2], introduced the usage of context-dependent pronouns. Using pronouns in dictating programming
reduces repetitions of long expressions, which generally appear in typing programming.
However, dictating programming is still far fromfeasible to be used eﬃciently. We think one of thediﬃcult points that cause this is the noise in theinput. The existing designs of dictating programming are focusing on making the grammar easier
to dictate, or implementing simple voice macros toachieve more complicated actions by speaking lesswords. They assume the input text is the alreadyrecognized perfect text without any error.
Some existing works such as VoiceCode [4] doconsider the incorrect recognition by providing thevoice macro to retreat the user’s last speaking,which indirectly solves the problem by increasing