日本ソフトウェア科学会第 38 回大会 (2021 年度) 講演論文集

A Preliminary Design of an Easy-to-Dictate
Programming Language with Pronouns
Wennong Cai

Soramichi Akiyama

Shigeru Chiba

Programming-by-voice is a technique that enables programmers to code by partially or fully using voiceinput. Many existing researches attempt to design an easy-to-dictate spoken programming syntax that canmap into the syntax of an existing typing programming language. However, they are still thinking in atyping programming style, while dictating should have its own advantage over typing. This paper proposes
the usage of context-dependent pronouns in the programming-by-voice design. The goal of using pronouns
is to replace long repetitive information with a short-length pronoun. In typing programming, repetitive
information can be typed quickly through the support of auto-complete, while the spoken programming canhardly have such support. This paper presents our prototype spoken programming language: NPLang, andshows the mechanism of using pronouns inside NPLang. In the end, this paper analyzes the limitation ofcurrent proposed design and concludes with future direction.

board requires 3 times click: convert to numerical

1 Introduction

mode button, convert to punctuation mode button,

Programming supports various essential parts inand ﬁnally click on the target key.

human’s daily life, while the request of ﬁxing code

Using voice-input as an assistance tool to support

might occur anytime for a programmer. However,programming on small devices will alleviate suchwhen people need to go for a business trip, it is in-diﬃculty. Programming-by-voice is a research topic

convenient to carry a laptop only for preparing any

that attempts to combine programming and voice,suddenly appeared coding work. On the contrary,

though some researchers in this area are motivated

it is ineﬃcient to complete coding work if they can-from diﬀerent aspects such as helping Repetitive

not exploit spare time during the trip. Therefore,

Strain Injury programmers [2] [3] [4].

being able to do programming on phones or tablets
is a lightweight and eﬃcient solution.

While existing programming-by-voice designs arestill not good enough due to their long length, weHowever, it is diﬃcult to do programming onshow our idea of using pronouns in the program-small devices because of the small screen size.

ming to refer existing long-length identiﬁers or ex-

For example, “+”, “=” or “{ }” are frequently

pressions in order to reduce length by pruning re-used punctuation marks in most programming lan-dundant information. We also present the proto-guages, but typing either of them on an iPhone key-

type programming language NPLang we have im-∗ 代名詞を使った口述プログラミング言語の初期設計

This is an unrefereed paper. Copyrights belong tothe Author(s).
蔡 文農, 穐山 空道, 千葉 滋, 東京大学大学院情報理工学系研究科, Graduate School of Information Science andTechnology, The University of Tokyo.

plemented that makes use of our idea.
In this paper, we ﬁrst introduce the backgroundof programming-by-voice by presenting some existing programming-by-voice researches. Then we
present our proposal of using context-dependentpronoun and its usage examples inside our prototype programming language. At last, related work

34

currRecNum = 10
database = [ maxOffSet , prevRecNum ,currRecNum ]of using pronouns in programming language will beintroduced before the conclusion.

2 Programming-by-Voice Background
There are several existing researches that attempts to design a programming-by-voice system.
However, they are not easy-to-dictate enough, because they only focus on designing the syntax for
programming language on a context-independentstyle, which means dictating each statement istreated independently.
For instance, VoiceGrip [3] is a system that translates a spoken programming language text into theC language code. It provides some pre-deﬁned syntax structures so that users could dictate easierthan reading the C code directly. A usage example
is as follows:then”

The algorithm used by VoiceGrip and VoiceCode
could help users easily dictating the three camelCase variables in the ﬁrst three lines, however,those variables appear again in the fourth line.
Users cannot make use of the 3 sentences they havedictated previously to support the dictation of thefourth line, but have to speak “max oﬀ set [pause]
previous record number [pause] current record number ” again, which is an ineﬃcient and unpleasantexperience. Dictating longer also means harder forthe speech recognition system to fully and accurately recognize user input.
We consider those repetitively appeared information as redundant information in the spoken pro-the experience of spoken programming, but does
not cause a serious problem in typing program-

will generate the code:

3

Variables occur repetitivelygramming context. Redundant information limits“ if current record number less than max oﬀ set

1
2

Code 2if ( currRecNum < maxOffSet ) {|// cursor will a u t o m a t i c a l l ymove to here}
Code 1

VoiceGrip example

ming because there are existing mature tool-levelsupports such as the auto-complete feature. Autocompletion helps users quickly typing long variable names appeared previously, while the spoken programming is currently lack for good-quality
auto-complete support. One of the reasons that

The VoiceGrip designs an abbreviation algorithm

makes auto-complete not suitable for spoken pro-to combine multiple words into a camelCase vari-gramming is the possible latency of voice recog-

able name, which rescues users from dictating every

nition system. For instance, to dictate the vari-character of a variable name that cannot be recog-

able name “currRecNum” with the help of auto-

nized by the speech recognition as a single word di-complete, a likely workﬂow might require speaking

rectly. Based on VoiceGrip, a further implementa-“current [wait speech recognition system to process,tion VoiceCode [4] has designed more voice macros

and editor to propose several completion candidates]

to support navigating cursors by dictating.

ﬁrst option”, which makes dictation more com-However, those systems are still designing theirplicated than only speaking “current record num-syntax in a typing programming thinking. For ex-ber ”. Therefore, redundant phrase is a diﬃculty

ample, to dictate the python code in Code 2:

that makes dictation harder.

1
2

maxOffSet = 30
prevRecNum = 20

Figure 1

NPLang User Interface. The keyword ‘them’ refers to multiple variables: “maxOﬀSet”,“prevRecNum”, “currRecNum”

than the phases it is referring to, which is the nor-3 Proposal: Context-Dependent Pronoun

mal case in either natural language or our prototypelanguage. If those identiﬁer names or expressions

3. 1 Overview

are used repetitively and intensively, the sentence

We propose the idea of Context-Dependent Pro-

that needs to be spoken becomes verbose and con-noun, and present how it prunes the redundant in-tains many repetitive phases if we do not introduce

formation from the spoken programming code.

context-dependent pronouns.

A context-dependent pronoun is a pre-deﬁned
keyword embedded in the spoken programming
syntax, which refers to existing identiﬁers or ex-

3. 2 Prototype Programming Language:NPLangpressions based on previous code context. Those

The NPLang (Natural Programming Language)

pronouns are replaced by the phases they are refer-

is a programming language we have designed forring during the compilation to form a valid code ofour pronouns experiment. Its syntax is designed

an existing typing programming language.

to be similar to English so that it will be easy-

For instance, in the previous Python example

to-dictate. The NPLang compiler we have imple-

Code 2 mentioned in Section 2, users have to speakmented converts the NPLang code (as text form)

9 words to dictate all elements in the list, and more

into the equivalent Python code.

words will need to be spoken if the length of list in-

In NPLang, we focus on embedding the con-creases. In our system, as Figure 1 shows in the

cept of context-dependent pronouns into the syn-fourth line, users only use one keyword ‘them’ totax, therefore we assume the input variable namerefer all three variables. The python code compiled

is already well-recognised, instead of reproducing

from it is the same as Code 2.

the algorithms for recognising abbreviated variable

Since the meaning of the keyword ‘them’ changes
depending on the previous code context, we nameit the context-dependent pronoun.

name from the work of VoiceGrip [3].
The BNF syntax is showed in Figure 2. The terminal symbol “TYPE” is used to distinguish string

The context-dependent pronouns prunes the re-data from other types of data as well as the iden-dundant information when a pronoun is shorter

tiﬁer, so that users do not have to dictate “openFigure 2BNF syntax of NPLang

double quotes” explicitly. Our current system does

based on its previous context. In current design,

not support loop statement or conditional state-

the meaning of each pronoun in NPLang is resolved

ment since we only focus on embedding the con-through a switch statement, and match diﬀerent

cept of pronouns into spoken programming at thecases based on the category of previous statements.

current stage.
The core point of NPLang’s syntax is that theprogram consists of multiple sentences, and eachsentence consists of multiple diﬀerent kinds ofstatements concatenated by the keyword ‘then’.
The context-dependent pronoun we propose works
in statement-level, which means each sentence isSpeciﬁcally, the singular pronoun ‘it’ is interpreted as follows based on the category of the closest previous statement:• Declaration: refer to the declared variable
name.
• Assignment: refer to the assigned variablename.

considered as an independent context, and the• Calculation: refer to the assigned variable

meaning of each pronoun depends on previous

name if it involves calculation assignment op-statements within the same sentence.

erator ‘by’ (i.e. +=, -=, *=, /=), else refer tothe whole expression.

3. 3 Rules of Analyzing Pronouns• Invoke: refer to the whole expression.

During the compilation, each context-dependent• Function:

pronoun is replaced by the phrases it is referring

name.

refer to the declared function

For the plural pronoun ‘them’, all previous state-3}

ments within the same sentence are examined sepCode 4

arately through the above process, then separate

Refer Expression

In the current design, parameters of a function

results are concatenated with the comma separa-

are automatically named “p” + the sequence num-tor.

ber of parameters. The pronoun ‘it’ in this example
refers to the expression “add p1 and p2”.

3. 4 Usage Examples
3. 4. 1 Refer based on Context

3. 4. 3 Expression Omission

Figure 1 has already shown that using theTo make the generated Python code more con-context-dependent pronoun could refer to a sin-cise, the NPLang compiler will omit an expression

gle or multiple identiﬁer name(s).

if it is being referred later and has no necessary

As our pro-posal name “context-dependent” suggests, the en-side-eﬀect such as assignment.

tities being referred depend on previous program-

For instance, in the previous example Code 4, in-ming context, as shown in Code 3. The ﬁrst oc-

side the deﬁnition of function “addTwo”, there are

currence of keyword ‘it’ refers to ‘num’, while thetwo statements in the spoken text, which are a cal-second one refers to ‘greeting’.

culation, and a return statement with a pronoun

1‘it’. The compiler ﬁrst translates them respectively

2
34

12
3
4

set variable num with integer value10 ,then add it by integer value 5.
set variable greeting with stringvalue hello ,then invoke print with it .

into “p1+p2” and “return p1+p2”, then the former
statement is omitted since it has been referred by
the latter statement and has no assignment eﬀect.
Such design avoids the generated Python code

Compiled python result:becoming verbose or unnatural. The example Codenum = 10
num += 5
greeting = " hello "
print ( greeting )5 proves this more obviously, where all of the fourstatements are included in one single statementwithout changing program logic.
1Code 3

Refer based on Context
2

3. 4. 2 Refer Expression
As described in the rules of analyzing pronouns,

3
4

add integer value 1 and integer
value 2 ,then multiply integer value 2 andinteger value 5 ,then invoke addTwo with them ,then invoke print with it .

a pronoun sometimes refers to an expression rather
Compiled python result:than the identiﬁer name in order to be more easyto-dictate.
12
3
4

define a function addTwo taking 2
arguments .
add p1 and p2 ,then return it .
end define .

Compiled python result:

1print ( addTwo (1 + 2 , 2 * 5) )Code 5

Expression Omission

3. 5 Limitation
Currently we use pre-deﬁned rules to resolve themeaning of each pronoun in the NPLang code.

12

def addTwo ( p1 , p2 ) {return p1 + p2

Also, as shown in previous sub-sections, the pronoun in our preliminary design has only two op-

tions, which is either ‘it’ for singular or ‘them’ forin our research, the context-dependent pronoun

plural.

serves the purpose of being an alternative but sim-Such design is ineﬃcient if we introduce newpronoun words apart from ‘it’ and ‘them’ in fu-

pler way to express existing identiﬁers or expressions.

ture. Because adding every new pronoun word re-Another programming language example of using

quires more delicately designed rules to avoid caus-

pronoun words is AppleScript [1]. In AppleScript,

ing grammar ambiguity.

the keyword ‘me’ refers the current script and theAnother limitation of the current design is thatkeyword ‘it’ refers to the current target. As Code 7

users have to remember those pre-deﬁned rules in-(cited from [1]) shows, the word ‘its’ and ‘my’ arestead of just using them intuitively in order toused to access properties of the objects referred touse context-dependent pronouns correctly, whichby ‘it’ and ‘me’.
Similar to previous example, AppleScript only

increases the learning curve of such technique.

provides pronouns for referring the current scope

4 Related Workitself, rather than referring entities appeared in theSome pronoun words are already widely known asprevious context.

Our system exploits pronoun

keywords in programming languages. For instance,words in a way that more similar to the usage inthe keyword “this” in well-known in some program-natural languages.

ming languages such as Java [5] or JavaScript [6]. It

1
2
3

is usually used to refer the current object scope inorder to fully exploit the design of object-oriented
programming. For instance, without the help of
the keyword ‘this’, the following JavaScript code

45becomes verbose.
1
2
3
4
5
6
78
91011121314var a = {v: 1,setV : function ( n ) {this . v = n ;}}var b = {v: 1,setV : function ( obj , n ) {obj . v = n ;}}a . setV (0) ;// with ‘ this ’b . setV (b , 0) ; // without ‘ this ’Code 6Keyword ‘this’ in JavaScript

678
910111213tell application " Finder "version-- output : "10.5.1" ( Finder
version is the default
in tell block )its version-- output : "10.5.1" (specifically asks for
Finder version )version of me-- output : "2.0" ( AppleScriptversion )my version-- output : "2.0" ( AppleScriptversion )version of AppleScript-- output : "2.0" ( AppleScriptversion )end tell
Code 7

AppleScript Example

5 Conclusion

The primary goals of the pronoun keyword ‘this’In this paper, we introduced the background of

from JavaScript and the Context-Dependent Pro-

programming-by-voice and our proposal of embed-

noun in NPLang are diﬀerent. In OOP languages,

ding context-dependent pronouns to prune the re-

keywords such as ‘this’ or ‘self’ are generally used todundant information in order to make the spoken

enhance the usage of object-oriented design, while

programming syntax easier to be dictated.

Wealso presented our prototype programming lan-knowledge understanding of the pronouns.

guage NPLang, and showed how our proposal perform in this prototype.
We have mentioned that the current design suffers from the limitation of ﬂexibility to design more
pronoun words, as well as the eﬃciency for users
to learn how to use context-dependent pronouns.
Therefore, as the future direction, we are considering to apply coreference resolution models fromthe area of Natural Language Processing into theNPLang in order to resolve the pronoun keywords
automatically. The coreference resolution modelis used to group phrases that are referring to thesame entity. It is a evolving machine learning solution to automatically resolve the meaning of pronouns in natural languages. The usage of spoken
programming language might be more ﬂexible withhelp from such model, because the model might beable to infer the referred phrases from the semantic

参 考 文 献[1] AppleScript Language Guide, Apple: The it andme Keywords. Retrieved from: https://developer.
apple.com/library/archive/documentation/AppleSc
ript/Conceptual/AppleScriptLangGuide/conceptual/ASLR fundamentals.html.
[2] Arnold, S. C., M. L. and Goldthwaite, J.: Programming by voice, VocalProgramming, In Proceedings of the fourth international ACM conference on
Assistive technologies, 2000, pp. 149–155.
[3] Desilets, A.: VoiceGrip: a tool for programmingby-voice, International Journal of Speech Technology, Vol. 4, No. 2(2001), pp. 103–116.
[4] Desilets, A., F. D. and Norton, S.: Voicecode:
An innovative speech interface for programmingby-voice, In CHI’06 Extended Abstracts on Human Factors in Computing Systems, (2006 Apirl),
pp. 239–242.
[5] Java Documentation, Oracle: Using the this Keyword. Retrieved from: https://docs.oracle.com/java
se/tutorial/java/javaOO/thiskey.html.
[6] JavaScript Reference, MDN Web Docs: This. Retrieved from: https://developer.mozilla.org/docs/
Web/JavaScript/Reference/Operators/this.