Learning to Find Naming Issues withBig Code and Small Supervision
Jingxuan He

Cheng-Chun Lee

ETH Zurich, Switzerland
jingxuan.he@inf.ethz.ch

EPFL, Switzerland
cheng-chun.lee@alumni.epfl.ch

Veselin Raychev

Martin Vechev

Snyk, Switzerland
veselin.raychev@snyk.io

ETH Zurich, Switzerland
martin.vechev@inf.ethz.ch

Abstract

1

We introduce a new approach for finding and fixing naming
issues in source code. The method is based on a carefulcombination of unsupervised and supervised procedures: (i)unsupervised mining of patterns from Big Code that express
common naming idioms. Program fragments violating suchidioms indicates likely naming issues, and (ii) supervised
learning of a classifier on a small labeled dataset which filters
potential false positives from the violations.
We implemented our method in a system called Namerand evaluated it on a large number of Python and Java programs. We demonstrate that Namer is effective in finding
naming mistakes in real world repositories with high precision (∼70%). Perhaps surprisingly, we also show that existing deep learning methods are not practically effective andachieve low precision in finding naming issues (up to ∼16%).

Names in source code are important as they often convey program semantics and capture developers’ intention, helping
with tasks such as code understanding [8], completion [7],and debugging [38]. Names also play a key role in code quality and software maintenance [18], while some naming problems are misuses and even bugs [9, 28, 41]. The importance ofnames is reassured by many works in academia [30, 34] andindustry [41]. Thus, a system able to automatically detectand fix naming issues is highly desirable.

CCS Concepts: • Software and its engineering → Software defect analysis; • Theory of computation → Program analysis.
Keywords: Name-based program analysis, Static analysis,
Bug detection, Anomaly detection, Machine learning
ACM Reference Format:
Jingxuan He, Cheng-Chun Lee, Veselin Raychev, and Martin Vechev.
2021. Learning to Find Naming Issues with Big Code and Small
Supervision. In Proceedings of the 42nd ACM SIGPLAN International
Conference on Programming Language Design and Implementation(PLDI ’21), June 20–25, 2021, Virtual, Canada. ACM, New York, NY,
USA, 16 pages. https://doi.org/10.1145/3453483.3454045
Permission to make digital or hard copies of all or part of this work forpersonal or classroom use is granted without fee provided that copies
are not made or distributed for profit or commercial advantage and that
copies bear this notice and the full citation on the first page. Copyrights
for components of this work owned by others than the author(s) must
be honored. Abstracting with credit is permitted. To copy otherwise, or
republish, to post on servers or to redistribute to lists, requires prior specificpermission and/or a fee. Request permissions from permissions@acm.org.
PLDI ’21, June 20–25, 2021, Virtual, Canada© 2021 Copyright held by the owner/author(s). Publication rights licensed
to ACM.
ACM ISBN 978-1-4503-8391-2/21/06. . . $15.00
https://doi.org/10.1145/3453483.3454045

Introduction

Key challenge. While important, detecting and fixing
naming issues in programs is a difficult task. Conventional
methods such as abstract interpretation [21], symbolic execution [19], or testing [1], do not capture names and thus failto detect naming issues. Indeed, since identifier names arecloser to natural language, differentiating between good andbad names inherently requires statistical reasoning [13, 40].
Statistical naming issue detectors are ideally built on a large
scale supervised dataset where real programs are labeledwith having naming issues or not. However, to the best of
our knowledge, such a large dataset does not exist and is
hard to obtain with either manual or automatic approaches.
To address this challenge, existing works rely on either
generating synthetic mistakes or on methods where labels
are not required. Deep neural networks are currently trained
and tested on large labeled datasets constructed by injecting synthetic defects in programs [9, 28, 45, 47]. However,despite achieving high test accuracy, we found these methods to have low precision in finding real world issues (discussed in § 5.6). While initially surprising, further thought
reveals that the issue is caused by the fundamental problem
of distribution mismatch: the distribution induced by generating synthetic defects does not match the distribution ofreal world mistakes. Alternatively, anomaly detection based
approaches do not require labeled data but often require substantial manual effort for creating effective rules and tuning
appropriate thresholds [34, 38, 41]. As a result, these systems
are often limited to certain types of bugs (e.g., wrong usagesof argument names) and language (e.g., Java).

PLDI ’21, June 20–25, 2021, Virtual, Canada

Jingxuan He, Cheng-Chun Lee, Veselin Raychev, and Martin Vechev

2

Small data[labeled]

Big code[unlabeled]Learning
NamingIssue & Fix
ProgramName patterns

Classifier

Figure 1. Our recipe for finding and fixing naming issues.
This work. In this work, we propose a new recipe forlearning to detect naming issues. The key idea is to split thelearning procedure into two steps, see top of Figure 1: (i)mine name patterns, interpretable naming rules that capture
a diverse set of naming idioms, from a large dataset of programs (i.e., Big Code), and (ii) train a binary classifier thatuses high-level expressive features and predicts whether aviolation of a name pattern is an issue. Importantly, step (i)does not require programs to be labeled with having issues
or not, while step (ii) only requires a small labeled dataset,
which is feasible to produce manually. This combination obviates the need to provide synthetic data and ensures that
one trains and tests on the same real world data. To use
the method at inference time, we follow the steps shownat the bottom of Figure 1. Given a program fragment (with
its representation): (i) we query the mined name patterns
to check if the fragment violates any of them, and (ii) if it
does, we query our learned binary classifier. A naming issue
is reported only if the classifier predicts the violation to betrue. The suggested fix is to change the relevant parts of thefragment so the originally violated pattern is satisfied.
We implemented our recipe in a system called Namerand evaluated it on both statically typed (Java) and dynamically typed (Python) languages. We show that Namer issignificantly more effective at finding naming issues thanstate-of-the-art deep networks trained on synthetic defects.
Main contributions. Our key contributions are:
• A novel method for detecting and fixing naming issues incode consisting of interpretable name patterns mined fromBig Code which capture a diverse set of naming idioms(§ 3.2, § 3.3), and a machine learning classifier with expressive, high-level features to filter false positives (§ 4.2).
• An end-to-end implementation of our method in a tool
called Namer. Namer supports Python and Java, leverages
powerful static analyses for extracting semantic information, and is readily applicable to other languages (§ 5.1).
• An extensive evaluation of Namer on a large, real world
dataset containing millions of files from GitHub, demonstrating that Namer achieves high precision (∼70%) and ispractically effective (§ 5). Our evaluation also shows that
despite being accurate on programs with synthetic defects,
current deep learning techniques [9, 28] achieve very low
precision (i.e., up to ∼16%) on detecting real issues (§ 5.6).

Overview

In this section, we provide an overview of Namer on an
illustrative example. We run the inference pipeline in Figure 1
to show how Namer finds and fixes a naming issue.
Example program. In Figure 2 (a), we present a code snippet taken from an open-source Python project hosted on
GitHub1 . This snippet defines a class TestPicture inheritingfrom TestCase, which is from the unittest library and isused for representing test units. The TestPicture class contains a function test_angle_picture that calls assertTrue, afunction from the parent class TestCase, with two arguments
picture.rotate_angle and 90. We underline the describedcall statement in the figure.
The use of assertTrue in Figure 2 (a) is incorrect. Based on
the documentation [6], the second argument of assertTrue
is an optional error message to display instead of a value tocompare to. Therefore, we can infer that this code calls thewrong API function, which will cause unexpected behavior
at runtime. Based on the code context, the correct function
to call is assertEqual, which checks whether the values ofthe first two arguments are equal. Namer finds this bug andsuggests to replace assertTrue with assertEqual. Next, werun the pipeline in Figure 1 on the buggy program statementand describe in detail how Namer finds and fixes this bug.
Program statement to AST+. Namer first parses the input program and obtains an abstract syntax tree (AST) for
each program statement. Figure 2 (b) shows the AST forthe buggy statement in our example with the nodes for
picture.rotate_angle omitted for simplicity. Then the following transformations are applied on the parsed tree toobtain a transformed tree (AST+) as shown in Figure 2 (c):1. Replace integer values (i.e., 90) with a special token NUM.
2. Add a special node NumArgs(2) as the parent of the function call node Call, showing that the function call has twoarguments (i.e., picture.rotate_angle and NUM).
3. Split each terminal node into subtokens based on standard
naming conventions and insert a node NumST(k) where kis the number of subtokens in the original node. In theexample, assertTrue is split into assert and True, andNumST(2) is inserted as a parent of the subtoken nodes.
The nodes with value NumST(1) are added for node self
and node NUM because they are unsplittable and contain
only one subtoken.
These transformations allow Namer to detect naming issues
on the subtoken level and make decisions based on additional
information such as the number of function arguments.
Then, Namer applies an additional semantic transformation. Namer runs interprocedural points-to and data flow
analyses which track the origin of each object and decorates
1 https://github.com/paulhildebrandt/python-keynote/blob/master/tests/

test_keynote_api.py

Learning to Find Naming Issues with Big Code and Small Supervision

PLDI ’21, June 20–25, 2021, Virtual, Canada
NumArgs(2)class TestPicture ( TestCase ):... // other functions in the classdef test_angle_picture ( self ):rotated_picture_name = " IMG_2259 . jpg "for picture in self . slide . pictures :if picture . relative_path \== rotated_picture_name :picture = self . slide . pictures [0]self . assertTrue ( picture . rotate_angle , 90)
break

Call
AttributeLoad
NameLoad
self

...

Attr

Call
...

AttributeLoad

Num

Num

NameLoad

Attr

NumST(1)NumST(1)NumST(2)NUM

90assertTrue(b) A parsed AST.

TestCase

TestCase

TestCase

self

assertTrue(c) A transformed AST (AST+).

(a) An example Python program from GitHub.
Condition:
NumArgs(2) 0 Call 0 AttributeLoad 0 NameLoad 0 NumST(1) 0 TestCase 0 self

NumArgs(2) 0 Call 0 AttributeLoad 0 NameLoad 0 NumST(1) 0 TestCase 0 self

NumArgs(2) 0 Call 0 AttributeLoad 1 Attr 0 NumST(2) 0 TestCase 0 assert

NumArgs(2) 0 Call 0 AttributeLoad 1 Attr 0 NumST(2) 0 TestCase 0 assert

NumArgs(2) 0 Call 0 AttributeLoad 1 Attr 0 NumST(2) 1 TestCase 0 True

NumArgs(2) 0 Call 2 Num 0 NumST(1) 0 NUM

NumArgs(2) 0 Call 2 Num 0 NumST(1) 0 NUM

Deduction:...

NumArgs(2) 0 Call 0 AttributeLoad 1 Attr 0 NumST(2) 1 TestCase 0, Equal(d) Name paths extracted for the transformed AST in Figure 2 (c).
(e) A name pattern violated by the name paths in Figure 2 (d).

Figure 2. Overview of Namer on an example program.
the AST with semantic information obtained from the analyses results. For our example program, the analyses identify
that the origin of the object self is TestCase. Namer thenincorporates the analysis results in the AST. In Figure 2 (c),Namer inserts nodes with value TestCase (filled with green )as the parents of node self, assert and True. The analyses
are described in § 4.1. We show in § 5 that running the analyses is a key factor for Namer to achieve high precision.
AST+ to name paths. Namer then traverses the transformed AST in a top-down fashion to extract name paths, ourprogram abstraction for identifier name usages. A name path
represents a path from the tree root to a leaf subtoken. In
Figure 2 (d), we show a list of name paths extracted from thetransformed AST in Figure 2 (c). Each name path contains
two parts. The first part is a list where each element consists of a node and the index of the next node in the current
node’s children list. The second part of a name path is a leaf
subtoken. For instance, the first line represents the namepath from the tree root to the leaf self. Along this path, thefirst child is always selected so the indices are all zero. We
formally define name paths in § 3.1.
Compared to existing syntactically constructed path abstractions [11–13, 33], our name paths contain richer semantic information from the transformed ASTs and have finer
granularity as they can be used for modeling subtokens.
Pattern matching. The extracted name paths are thenchecked against a set of interpretable naming rules called
name patterns. The name patterns are automatically minedfrom a large dataset of open source repositories and capture
a wide range of common naming idioms. If the name paths
violate any of the name patterns, then the statement deviatesfrom common naming behaviors and a potential naming
issue is detected by Namer. Moreover, the violated namepattern suggests how to fix the issue: modify the statement
so that the violated pattern becomes satisfied. We formallydefine name patterns in § 3.2 and describe the mining algorithm in § 3.3. In Figure 2 (e), we show an example namepattern which consists of two sets of name paths, the condition and the deduction. At a high level, the name patternindicates if a program statement calls a function that satisfies
the following three requirements, as specified by the threename paths in the condition:
1. the receiver is self and is an instance of TestCase, and2. the first subtoken of the function name is assert, and3. the second function argument is a numerical value,then the second subtoken of the function name should be
Equal, as specified by the name path in the deduction. Our
example statement self.assertTrue(picture.rotate_angle,90) violates the name pattern in Figure 2 (e) because its namepaths satisfy the condition but contradict with the deduction.
The violation exhibits a potential naming issue and the fix
suggested by Namer is to replace True with Equal.
Classifying violated patterns. So far, the mined namepatterns can already identify anomalous naming behaviors
and report them as naming issues. Balancing the tradeoff
between recall and precision is known to be difficult for
anomaly detectors [38, 41], especially when our mining process is performed on millions of source files. As a result, withdifferent choices of hyperparameters in the matching andmining processes, the mined patterns will either find very
few issues because the violations are rarely triggered, or will

PLDI ’21, June 20–25, 2021, Virtual, Canada

inevitably have lower precision because more violations are
triggered on less anomalous code.
In Namer, we allow violations to be triggered at lower
confidence so that most issues are not missed. Then, to prune
false positives from the triggered violations, we build a binary
defect classifier that uses various statistical measures fromthe violations as features. The classifier outputs true if thereport should be reported to the user as a naming issue,or false otherwise. We describe the classifier in § 4.2. Ourevaluation in § 5 shows that the classifier is a crucial element
for Namer to achieve a high precision.
Issues handled by Namer. Namer targets a wide rangeof program elements (e.g., variables, functions, API calls,
types, etc.) and usually reveals issues (e.g., indescriptive
names, wrong API usages, wrong types, etc.) where thenames of the elements do not correctly capture the underlying code semantics. On the AST level, Namer targets all
terminal nodes with code names. Examples of naming issues
detected by Namer are shown in Tables 3 and 6. In Figure 2, we show a semantic defect that causes an unexpected
program behavior. The vast majority of Namer’s reports
are code quality issues that may impair code readability andmaintainability.

3

Program Abstraction for Names

In this section, we describe our program abstractions forcode names. We start by defining how we augment abstract
syntax trees (ASTs) with results of program analyses andthen how we extract name paths from these ASTs. Next,
we define name patterns for representing common usagesof identifier names. Finally, we present the algorithms formining name patterns from a large dataset of code.
3.1

Abstract Syntax Tree and Name Path

We start by formally defining ASTs for program statements.
Intuitively, this is part of the abstract syntax tree of the whole
program, projected on a specific statement only.
Definition 3.1 (Abstract Syntax Tree). An Abstract Syntax
Tree (AST) for a program statement is a tuple ⟨N ,T , r, δ, V , ϕ⟩,where N is a set of non-terminal nodes, T is a set of terminal nodes, r ∈ N is the root node, δ : N → (N ∪ T )+ is afunction mapping a non-terminal node to the list of childnodes, V is a set of node values, and ϕ : (N ∪ T ) → V is afunction mapping a node (either terminal or non-terminal)
to its associated value. A parsed AST is shown in Figure 2 (b).
Given a parsed AST, Namer performs the following steps
to obtain a transformed AST:
1. Transform all numerical values into a special token NUM,
all string values into STR, and all boolean values into BOOL
to improve generalization.
2. For each function definition or function call node n, insert
a node n ′ with value NumArgs(k) between n and n’s parent,

Jingxuan He, Cheng-Chun Lee, Veselin Raychev, and Martin Vechev

where k is the number of arguments in the definition or
the call.
3. For each terminal node n whose value is an identifier
name, split the name into subtokens based on standard
naming conventions such as camelCase and snake_case.
Then, replace n by a non-terminal node n ′ with value
NumST(k), where k is the number of produced subtokens.
The subtokens are appended as n ′’s children.
4. For each terminal node representing an object name, insert
a node whose value is the origin of the object as the parent.
For each terminal node representing a function call, insert
a node whose value is the origin of the receiver object (ifany) as the parent. The information about object origin
is obtained via interprocedural points-to and data flow
analyses described later in § 4.1.
Note that due to variadic or keyworded arguments, calls
to the same function may have a different number of arguments. Such cases are captured by the AST and are nottreated specially. The pattern mining process in § 3.3 will automatically learn the behaviors of such functions. We show
an example of transformed ASTs in Figure 2 (c). Transformed
ASTs contain more fine-grained (e.g., subtokens), semantic
information (e.g., results from static analyses) than parsed
ASTs, which strengthens the precision of our name path andname pattern abstractions. Given a transformed AST, thenext step is to extract name paths defined in the following.
Definition 3.2 (Name path). A name path extracted froma (transformed) AST is a tuple ⟨S, n⟩. We call S the prefix
and n the end node. S is a list of non-terminal nodes along
the path from the AST root to a terminal node. Formally,S = [(n j , i j )]kj=1 where n j ∈ N is a non-terminal node in
the AST and i j ∈ N is an index, such that n 1 = r (n 1 is theroot node) and δ (n j )[i j ] = n j+1 (n j+1 is the i j th child of n j ).
Node n is either the concrete terminal node where the AST
path ends (δ (nk )[i k ] = n, i.e., n is the i k th child of nk ) or a
special symbolic node ϵ introduced for adding more degrees
of freedom to name patterns defined later in § 3.2. We callthe name path a concrete (resp., symbolic) name path if n is
concrete (resp., symbolic).
Example 3.3. In the following, we show three name paths
np1 , np2 and np3 . np1 and np2 are concrete name paths, and
np3 is a symbolic name path.
np1 : NumArgs(2) 0 Call 0 AttributeLoad 1 Attr 0 NumST(2) 1 TestCase 0 True
np2 : NumArgs(2) 0 Call 0 AttributeLoad 1 Attr 0 NumST(2) 1 TestCase 0 Equal
np3 : NumArgs(2) 0 Call 0 AttributeLoad 1 Attr 0 NumST(2) 1 TestCase 0 ϵDefinition 3.4 (Relational operators of name paths). Given
two input name paths np1 and np2 , we define the following
two relational operators:• np1 ∼ np2 : is true if np1 .S = np2 .S. That is, if np1 .S and
np2 .S have the same length, and all corresponding elements in np1 .S and np2 .S are equal; is false otherwise.

Learning to Find Naming Issues with Big Code and Small Supervision• np1 = np2 : is true if np1 ∼ np2 ∧ (np1 .n = ϵ ∨ np2 .n =ϵ ∨ np1 .n = np2 .n); is false otherwise.
Intuitively, np1 ∼ np2 requires that the prefixes of np1 and
np2 are the same. np1 = np2 additionally requires that theend nodes of np1 and np2 are equal, or either np1 .n or np2 .n
is ϵ (that is, any value is equal to ϵ).
Example 3.5. In Example 3.3, np1 ∼ np2 holds, np1 = np2
does not hold, and both np1 ∼ np3 and np1 = np3 hold.
Each name path captures one name in the AST and a
program statement can be represented by a set of name paths
extracted from its AST. An example of name paths extracted
from a code statement is shown in Figure 2 (d). The namepath representation makes it convenient to find common
properties of program statements for later construction ofname patterns, common coding idioms that generalize acrossstatements. We note that the set of name paths A extracted
from a program statement must satisfy two properties: ∀a ∈A. a.n , ϵ, i.e., all name paths in A are concrete, and ∀a 1, a 2 ∈A. a 1 .S , a 2 .S, i.e., all prefixes of the name paths in A are
different. These properties are useful when we check thename paths of a statement against name patterns.
3.2

Name Pattern

We now define name patterns which consist of name paths
and are used to represent the most common usage of identifier names in a statement.
Definition 3.6 (Name pattern). A name pattern p consistsof two sets of name paths, condition C and deduction D.
Intuitively, a name pattern requires that if a statement includes certain name paths (defined by C), then name paths
that conform to D should also be present in this statement.
We provide three relationships between a name pattern p
and a given statement s represented by a set of name paths
A: match, satisfaction, and violation. In the following, wedefine the match relationship and briefly introduce the othertwo at a high level. We formally define the satisfaction andviolation relationships individually for two specific patterntypes introduced later in this section.
• Match: s matches p if ∀c ∈ C. ∃a ∈ A. c = a and ∀d ∈D. ∃a ∈ A. d ∼ a, or intuitively, if all name paths in C
also exist in A and all prefixes of the name paths in D
also exist in A. For example, the name paths in Figure 2 (d)match the name pattern in Figure 2 (e). Note that there is
no requirement on the end nodes of name paths in the
deduction D.
• Satisfaction: s satisfying p means that s conforms to thenaming idiom represented by p.
• Violation: s violating p means that s violates the naming
idiom represented by p. Therefore, a potential naming
issue is revealed and the suggested fix is to modify thestatement so that its name paths satisfy p.

PLDI ’21, June 20–25, 2021, Virtual, Canada

The definition of name patterns is generic and allows one todefine different types of patterns to capture different kindsof naming idioms. Next, we introduce two types of name patterns in Namer while leaving the addition of more patterns
as future work items. The first one is consistency name patterns whose purpose is to ensure that code fragments withthe same underlying semantics should be named consistently,
which is an important aspect of code quality.
Definition 3.7 (Consistency name pattern). A consistency
name pattern requires that D = {d 1, d 2 } (i.e., the deduction
D consists of two name paths d 1 and d 2 ), and both d 1 and d 2
are symbolic. The satisfaction and violation relationships ofconsistency name patterns are defined as follows:• Satisfaction: s satisfies p if s matches p and ∀a 1, a 2 ∈ A.
(a 1 .S = d 1 .S ∧ a 2 .S = d 2 .S) =⇒ (a 1 .n = a 2 .n). Intuitively,
it requires that any two name subtokens in the statement
prefixed by paths d 1 and d 2 must be equal.
• Violation: s violates p if s matches p but does not satisfy p.
Example 3.8. In the following, we show a simple consistency name pattern which enforces that in a Python statement of the form self.<name1 > = <name2 >, the two names<name1 > and <name2 > must be equal.
Condition: Assign 0 AttributeStore 0 NameLoad 0 NumST(1) 0 Object 0 self
Deduction: Assign 0 AttributeStore 1 Attr 0 NumST(1) 0 ϵ
Assign 1 NameLoad 0 NumST(1) 0 Str 0 ϵ

The second type of name pattern used in Namer is theconfusing word name pattern. Its goal is to detect words thatdevelopers tend to mistakenly use and to suggest a correction.
To obtain such patterns, we extract the pairs of mistaken andcorrect words, referred to as confusing word pairs.
Confusing word pairs. A confusing word pair consistsof two words ⟨w 1, w 2 ⟩ where in some prior version of thecode, w 1 was used instead of w 2 . We call w 1 the mistaken
word and w 2 the correct word. We discover confusing wordpairs from all of the (deduplicated) commits in our dataset.
First, we apply a diff matching algorithm [37] over the ASTof the program before and after the commit. For each pairof matched nodes, the names are split into subtokens and if
there is one difference in the subtokens, the pair of subtokens
is added as a confusing word pair. We extracted 950K confusing word pairs for Java and 150K for Python. Examples ofmined confusing word pairs are ⟨name, key⟩, ⟨value, key⟩,⟨x, y⟩, ⟨min, max⟩, ⟨True, Equal⟩.
Definition 3.9 (Confusing word name pattern). A namepattern is a confusing word name pattern if D = {d} (i.e., thededuction D consists of a single name path d) and there exists
a pair ⟨w 1, w 2 ⟩ in the set of mined confusing word pairs suchthat d.n = w 2 . The satisfaction and violation relationshipsof confusing word patterns are defined as follows:

PLDI ’21, June 20–25, 2021, Virtual, Canada

Algorithm 1: Mining name patterns.
Procedure minePatterns(proдs, t)
Input :proдs, a set of programs.
t, the type of the name patterns to mine.
Output :patterns, a set of mined name patterns.
2
asts = getStmtAsts(proдs)
3tree = FPTree()4
for ast in asts do
5paths = getNamePaths(ast)6
for cond, deduct in splitPaths(paths, t) do
7tree.update(sort(cond) + sort(deduct))189patterns = genPatterns(tree.root, list(), t)
return pruneUncommon(patterns, asts)

Jingxuan He, Cheng-Chun Lee, Veselin Raychev, and Martin Vechev

Algorithm 2: Generating name patterns in FP tree.
Procedure genPatterns(node, paths, t)
Input :node, current node in the FP tree.
paths, a list of visited name paths.
t, the type of the name patterns to mine.
Output :patterns, a set of mined name patterns.
2
paths.append(node.path)3
patterns = set()4
if node.isLast then5deduct = getDeduction(paths, t)
6conds = getConditons(paths, t)
7
for cond in combinations(conds) do
8patterns.add((cond, deduct))1

11

for child in node.children() donewPatterns = genPatterns(child, paths, t)patterns = patterns ∪ newPatterns

12

return patterns

910• Satisfaction: s satisfies p if s matches p and ∀a ∈ A. a.S =d.S =⇒ a.n = d.n. Intuitively, it requires that the namesubtoken prefixed by the name path d in the deduction
must be equal to the correct word.
• Violation: s violates p if s matches p but does not satisfy p.
Figure 2 (e) shows an example of a confusing word pattern.
3.3

Mining Name Patterns from Big CodeAfter defining name patterns, the next question then is how
to obtain the most representative name patterns adopted bymost developers. We propose to mine name patterns froma large dataset of open source repositories, i.e., Big Code.
To this end, we propose an algorithm based on the frequent
pattern trees (FP trees) mining methods [24, 32]. The original procedures from [24, 32] treat all items the same. In
our scenario, name paths can be part of a condition or adeduction. Therefore, in our algorithm, we split name paths
into condition paths and deduction paths. At a high level,
the algorithm iterates through programs in the open source
repositories and extracts name paths to grow the FP tree.
The nodes of the FP tree stores the extracted name paths
and their occurrence counts. After the growth of the FP tree,
we traverse the FP tree to generate the most common namepatterns based on the frequency information.
The algorithm minePatterns for mining name patterns
is outlined in Algorithm 1. minePatterns takes a set of programs from open source repositories and the type of patterns to mine as input, and outputs a set of mined namepatterns. First, we obtain the transformed ASTs of the program statements in the repositories by parsing the programs
and applying the transformation rules (Line 2), and initialize
the FP tree (Line 3). Then, we iterate through the obtained
ASTs to update the FP tree (Line 4 to 7). For each AST, we
extract the name paths by calling getNamePaths at Line 5.
The name paths are then split (with the splitPaths function atLine 6) to condition (variable cond) and deduction (variablededuct). For the consistency name pattern, we treat pairsof name paths with the same end node as the deduction.
For the confusing word name pattern, we treat name paths
whose end node is equal to the correct word of a confusing
pair as the deduction. The name paths not treated as the
deduction are included in the condition. We iterate throughall possible ways to split (Line 6), put the sorted condition
and the sorted deduction into a single list, and update the
FP tree with the resulted list (Line 7). During the update, we
maintain the occurrence count of each node and set the flag
isLast (initialized to be false) of the last node in the input
list to true (not shown in Algorithm 1). The isLast flag isused later to determine when to generate name patterns.
After the FP tree is constructed, we traverse its nodes to extract name patterns (Line 8) by calling a recursive algorithm
genPatterns described in the next paragraph. Given the extracted name patterns (variable patterns), we further callpruneUncommon to prune uncommon patterns. To achievethis, we iterate through asts in the input dataset and countthe number of matches and satisfactions for each pattern.
We only keep the patterns whose ratio between the numberof satisfactions and the number of matches is above a certain threshold (we used 0.8), which means that the pattern iscommonly adopted in the dataset.
genPatterns is shown in Algorithm 2. It maintains a list ofvisited name paths in the function argument paths. First, thecurrent name path node.path is appended to paths at Line 2
and a set of mined patterns (variable patterns) is initialized
at Line 3. If the isLast flag of the current node is set to true(Line 4), we construct new name patterns. To achieve this,we need to assemble the deduction and condition of the
new patterns from paths. We obtain the deduction deduct(by calling getDeduction on paths at Line 5) and a set ofname paths conds that can be included in the condition (byLearning to Find Naming Issues with Big Code and Small Supervision
Condition

NP1 44NP2 33NP3 32
NP4 14NP5 15NP6 13(a) An example of FP trees.

NP1
NP1, NP3
NP1, NP3
NP1, NP3, NP4

PLDI ’21, June 20–25, 2021, Virtual, Canada

Deduction

Count

NP2
NP5
NP4
NP6

33
15
14
13

(b) Name patterns extracted bytraversing the FP tree example
in Figure 3 (a) with Algorithm 2.

Figure 3. Examples for our mining algorithm.
calling getCondtions on paths at Line 6). For the consistency
name pattern, we store the last two name paths of paths
in deduct and the other paths in conds. The end nodes of
the name paths in deduct are set to ϵ. For the confusing
word name pattern, we store the last name path of paths
in deduct and the other paths in conds. Next, we loop overall possible combinations of name paths in conds (Line 7)
where each combination can be seen as a valid condition, andadd the newly extracted pattern to the set of mined patterns(Line 8). The same procedure is performed recursively forthe descendants of the current node (Line 10).
We provide two examples below which illustrate the intermediate results of the algorithms.
Example 3.10 (A constructed FP tree). In Figure 3 (a), weshow a FP tree constructed after Line 7 of Algorithm 1 formining confusing word name patterns. Each node represents
a name path and shows its occurrence count. Green denotes
nodes with the flag isLast set to true.
Example 3.11 (Results of running Algorithm 2). In Figure 3 (b), we show all the extracted name patterns by running
genPatterns on the FP tree in Figure 3 (a). This result corresponds to variable patterns before Line 9 of Algorithm 1.

4

Index

Description

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

Number of name paths used to represent s
Number of statements identical to s on the file level
Number of statements identical to s on the repository level
Satisfaction rate of p on the file level
Satisfaction rate of p on the repository level
Satisfaction rate of p over the entire mining dataset
Number of violations for p on the file level
Number of violations for p on the repository level
Number of violations for p over the entire mining dataset
Number of satisfactions for p on the file level
Number of satisfactions for p on the repository level
Number of satisfactions for p over the entire mining dataset
Whether p targets on object name or function name
Number of name paths in p’s condition
Match ratio between p and s
Edit distance between the original name and the suggested name
Whether the original and suggested names form a confusing word pair

to return a fresh allocation site. As a result, the analysis is notalways sound, but this is not a requirement in our setting.
Using the results of the points-to computation, a dataflow
analysis is performed for primitive types. This allows us forevery variable, field, array access, and other action to obtain
possible origins. The origin of objects is their allocationsite. The origin of values is a function returning a valueor ⊤ if the value was modified after its creation. When theorigin sites are precisely computed (i.e. not abstracted to⊤), this information is added to the AST. This effectively
provides an abstraction where placing results of expressions
into temporary variables, extracting methods, or doing othersimple refactorings does not affect the augmented tree.

Detecting Naming Issues

In this section, we describe the points-to and data flow analyses for extracting the origins of objects, and the defectclassifier for pruning false positives.
4.1

Table 1. Extracted features for a violation consisting a statement s and a violated name pattern p.

Static Analyses for Name Path Context

To augment the ASTs for each statement in a program, Namer
statically analyzes each source file of the program. Every fileis analyzed in isolation, assuming every public method orfunction to be a possible entry point. Namer computes flowand context-sensitive Andersen style points-to analysis byusing k-call site sensitivity with k set by default to 5, unless
for a specific program this leads to a combinatorial explosionof more than 8 contexts per method on average. Practically,
this happens for a few programs in our large dataset. Our
points-to analysis is implemented in Datalog. We refer thereader to [44] for definitions of points-to analysis in Datalog.
Since our points-to analysis is performed on a per-file basis,any function or method defined outside the file is considered

4.2

Defect Classifier

A violation of the name patterns indicates a potential naming issue but can result in a false positive. This is because
when mining and matching name patterns, we increase thelikelihood to trigger more violations so that most issues are
not missed. Therefore, it is critical to prune false positives.
To this end, Namer feeds the violations of name patterns intoa machine learning procedure. Formally, given a program
statement s and a name pattern p, Namer invokes a feature
extraction function ϕ and a binary classifier ψ . Namer reports the violation consisting of s and p as a naming issue if
and only if ψ (ϕ(s, p)) = true. Intuitively, the violations classified by ψ as true are highly likely to be true naming issues sothey are reported. Those classified as false are recognized by
Namer as false positives so Namer abstains from reporting
them. ψ can be obtained by applying off-the-shelf learningalgorithms. To make the classification effective, the key thenis to extract a set of expressive features with ϕ.

PLDI ’21, June 20–25, 2021, Virtual, Canada

Features for violations. In Table 1, we show the list offeatures extracted by ϕ for a violation. In general, thesefeatures calculate the statistics of the violation and capture
the strength of the violation. In the following, we describe
the features one by one.
Feature 1-3 capture characteristics of the program statement s. Feature 1 calculates the number of name paths extracted from s. The more name paths s has, the more complex
s is. Feature 2 and 3 count the number of statements identicalto s in the file and the repository, respectively. More identicalstatements indicate a lower degree of anomaly.
Feature 4-14 measure various statistics of the violated pattern p. Feature 4-6 calculate the satisfaction rate (the numberof satisfactions divided by the number of matches) of p in
the file containing the statement s, in the repository containing s, and over the entire dataset used for mining the namepatterns, respectively. A higher satisfaction rate indicatesthat the naming idiom represented by p is more commonly
adopted. Feature 7-9 count the number of violations for p
and feature 10-12 count the number of satisfactions for p,both over three different levels. The larger feature 7-9 areor the smaller feature 10-12 are, the more common p is. Feature 13 is a boolean feature that checks whether p targets
an object name or a method name. The motivation here isthat during our early experiment, we found that most of thefalse positives came from name patterns targeting method
names as inferring correct usages of method names typicallyrequires more contexts. Feature 14 counts the number ofname paths in p’s condition. The more name paths in thecondition, the more difficult to match p.
Feature 15-17 jointly consider the statement s and theviolated pattern p. Feature 15 is the match ratio between pand s, i.e., the number of name paths in p’s condition divided
by the number of name paths in s minus the number of namepaths in p’s deduction. The higher the match ratio is, thebetter p captures the structure of s, and the more likely theviolation is a true naming issue. Feature 16 calculates the edit
distance between the original name that violates the pattern
and the suggested name. Smaller edit distance indicates ahigher probability of a true naming issue (e.g., typo). Feature
17 is a boolean feature that checks whether the original nameand the suggested name form a confusing word pair.
Unlike deep learning approaches [9, 28, 39] which use lowlevel embedding as initial features, all our features are highlevel, enabling our classifier to be trained with a small manually labeled dataset. Most of our features consider different
levels (i.e., file, repository, and the entire dataset), as opposed
to anomaly detection based approaches [34, 38, 41] which
rely only on a single or few measures. This is an importantdesign choice to make the classifier effective. As we showin § 5.5, the same feature can have different contributions to
the final decision over different levels.

Jingxuan He, Cheng-Chun Lee, Veselin Raychev, and Martin Vechev

5

Experimental Evaluation

The evaluation of Namer focuses on four main questions:• What is the precision of Namer on finding naming issues
in open source repositories?
• What is the impact of points-to and data flow analyses(§ 4.1) and the defect classifier (§ 4.2)?
• What features are most important in the classifier?
• How does Namer compare to existing deep neural network
based approaches?
5.1

Implementation and Evaluation Setup

Namer currently supports Python and Java though our framework is generic and can be applied to other languages.
Dataset. For both Python and Java, we obtained a large
real world dataset of open source repositories on GitHub [3].
The Python (resp., Java) dataset contains about 1 million(resp., about 4 million) source files from 33, 144 (resp., 33, 308)
repositories. We selected the set of repositories based on thehighest number of stars. For finding confusing word pairs,
we also used the entire histories of these repositories. Awareof code duplication on Github [35], we pruned our dataset
to make it free from project forks and file-level duplicates.
Mining name patterns. To obtain name patterns, we ran
the pattern mining algorithm introduced in § 3.3 over the entire dataset. The statistics of the mined patterns are discussed
later in the evaluation. To avoid overfitting and strengthen
generalization, we adopted standard techniques [12] to regularize name paths and patterns.
• We limited the number of name paths for a statement bysimply keeping the first 10 paths.
• At Line 5 of Algorithm 1, we only returned frequent namepaths that occurred more than 10 times in our dataset. As
a result, over 99% of infrequent name paths were removed.
• At Line 6 of Algorithm 2, we limited the maximal numberof name paths used in conditions to 10.
• At Line 9 of Algorithm 1, to prune uncommon patterns,
we only returned name patterns that occurred more times
than a given threshold. We found that setting the thresholdto 100 for Python (500 for Java) yielded good results.
Training the classifier. To train the classifier, we needa supervised dataset where violations are manually labeled
to true naming issues or false positives. In general, moretraining data can help the classifier perform better but thelabeling process requires expensive human labor. Since we
use simple models and high-level features, a small training
set should be sufficient. For both Python and Java, we manually labeled 120 violations, which strikes a good tradeoff
between precision and the amount of manual effort needed
for building the Namer tool.

Learning to Find Naming Issues with Big Code and Small Supervision

Among all labeled violations, one half were labeled true
and the other half were labeled false. We leveraged crossvalidation for model selection. We selected the support vector machine model with the linear kernel (the other choices
are logistic regression and linear discriminant analysis) as
it resulted in the best cross-validation results. We used feature standardization and principal component analysis as apreprocessing step for the features.
Testing and metric. At test time, we ran the patternmatching module of Namer over the entire Python and Javadataset excluding the samples used for training the classifier. For each dataset, we randomly selected 300 violations
and ran the trained classifier on the selected violations to
obtain the final reports of Namer. Then, we inspected all of
Namer’s bug reports and found that Namer is able to reportvarious kinds of naming issues. Following [39], we classify anaming issue into two categories:• Semantic defect: a naming issue is a semantic defect if thename in the evaluated program causes unexpected program behavior, is inconsistent with the program semantics,
or may introduce errors.
• Code quality issue: a naming issue is a code quality issue
if it is not a semantic defect but impairs code quality byconfusing the readers or is inconsistent with the namingstyle in the file. It is strongly recommended to fix thoseissues to improve code readability.
If a report does not belong to the above two categories, wemark it as a false positive. We provide examples of semantic
defects, code quality issues, and false positives in Tables 3
and 6. We calculate precision as the number of reported naming issues (including semantic defects and code quality issues) divided by the total number of reports. Note that unlike
standard bugs (e.g., buffer overflow), naming issues are lessstudied and their importance lacks common understanding.
Moreover, the severity of code quality issues can be subjective and varies among developers. To demonstrate that theissues reported by Namer are relevant, we conducted a smalluser study with professional developers, described in § 5.4.
Speed of Namer. The experiments were performed on a
28-core machine with two 2.60 GHz Intel Xeon E5-2690 CPUs
and 512 GB RAM running Ubuntu 18.04. The experiments
for [9] and [28] in § 5.6 require GPU so they were performed
on a machine with RTX 2080 Ti GPUs.
Namer is quite performant with its runtime dominated by
the program analyses described in § 4.1. On average, Namer
spent 20ms for Java and 39ms for Python analyzing one
source file on a single core of the test server. Because eachfile is analyzed separately, we parallelized the analysis on all
28 cores of the machine.

PLDI ’21, June 20–25, 2021, Virtual, Canada

Table 2. Precision of Namer and baselines on 300 randomly
selected violations from the Python dataset. "C" stands forthe defect classifier and "A" stands for the static analyses.
Baseline

Report

Namer
w/o C
w/o A
w/o C & A

134
300
88
300

5.2

Semantic Code quality
False
Precision
defectissue positive
5
13
2
12

89
124
50
108

40
163
36
180

70%
46%
59%
40%

Results for Python

We now evaluate Namer on the Python dataset. We first
discuss the precision of Namer. Then, we provide examplesof naming issues found by Namer. At last, we show detailed
statistics on the mined name patterns and the classifier.
Precision of Namer. For Python, we obtained 134 reports by running the trained classifier on the randomly selected 300 violations. The results of the manual inspection
are shown in the first row of Table 2. Among the reports
by Namer, we manually found 5 semantic defects, 89 code
quality issues, and 40 false positives, resulting in a precisionof 70%. The high precision demonstrates the effectiveness ofNamer on real world Python code.
Necessity of classifier and analyses. We now investigate the impact of the static analyses and the defect classifieron the effectiveness of Namer. To this end, we constructed
three baselines, by excluding the defect classifier, the analysis module, or both, from Namer. For each baseline, we
similarly inspected the bug reports on 300 randomly selectedviolations. The inspection results are reported in rows 2–4
of Table 2. Row 2 shows the results for the baseline where
the classifier was excluded from Namer, i.e., only patternmatching was performed. All 300 violations were reported
as naming issues where 163 were false positives, resulting ina precision of 46%. Row 3 shows the results for the baseline
for which the program analyses in § 4.1 were not performed.
Compared to Namer, this baseline found fewer semantic
defects and fewer code quality issues and was less precise.
From the comparison, we can see that the analyses andthe classifier are both necessary parts of Namer. The analyses not only improved Namer’s precision but also helped
Namer find more issues. The classifier may filter out sometrue positives but significantly reduced the number of false
positives, resulting in significantly higher precision while
retaining similar recall as before, which is a desired property in practice as it allows Namer’s users to spend minimal
efforts on inspecting the bug reports.

PLDI ’21, June 20–25, 2021, Virtual, Canada

Jingxuan He, Cheng-Chun Lee, Veselin Raychev, and Martin Vechev

Table 4. Manual inspection of 100 Python Namer reports
per pattern type with a breakdown of code quality issues.

Table 3. Examples of reports by Namer for Python.
Index

Reported statement

Suggested fix
Inspection outcome

Consistency

Confusing word

Semantic defect
Code quality issue
False positive

1
71
28

9
53
38

Semantic defects
1
2
3

self.assertTrue(vec, 4)

Equal

for i in xrange(10)range

self.assertEquals(3, val)

Equal

Code quality issues
45
6

num_or_process = 3
def evolve(self, ..., **args):
self.sz = N.array(sz)ofkwargsnp

False positives
7
8

self.assertTrue(os.path.islink(path))
self.read_line_block()

Breakdown of code quality issues
Confusing name19
Indescriptive name9
Inconsistent name26
Minor issue
15
Typo
2

14
6
23
7
3

existslog

Examples of reports made by Namer. We ran Nameron more statements in our Python dataset and present examples of reports in Table 3. For each example, we show thereported statement and the fix suggested by Namer:• Example 1 is similar to the one described in § 2 but the callhas different arguments. The issue may cause unexpected
runtime behavior. Namer correctly suggested to change
assertTrue to assertEqual.
• Examples 2 and 3 show the cases where Namer detects
and fixes deprecated API calls. In Example 2, xrange is abuilt-in function in Python 2 but gets removed in Python
3. In Example 3, assertEquals is a deprecated function in
the unittest library.
• Example 4 contains a typo. Namer proposed a correct fix
to the typo by changing or to of.
• Examples 5 and 6 are code quality issues where the original
names do not conform with the standard naming conventions. In Example 5, args is used for keyworded variable
length arguments. However, kwargs should be used instead for keyworded arguments and args should be used
for non-keyworded variable length arguments. Namer
suggested the exact fix for Example 5. In Example 6, N is
used as the abbreviation for the library numpy. However,np is commonly used and is a more informative name.
• Examples 7 and 8 are false positives of Namer. Example
7 asserts whether a given file path in the path variable
is a symbolic link by calling the function islink. However, Namer wrongly suggested to replace the function
with exists. This is because the usage of exists insideassertTrue is more frequent than the usage of islink in
our dataset. For Example 8, Namer suggested to replaceline with log according to the violated pattern. However,line is not an incorrect or confusing name according to
the semantics of the function (not shown here).

The above examples show that Namer can find and fix
different kinds of naming issues, including those that requirea deep semantic understanding of the program (e.g., correct
usage of API calls).
Statistics on pattern mining and classification. In total, 65, 619 name patterns were mined for the Python dataset.
496, 306 program statements, 439, 508 source files (50% of all
source files), and 30, 388 repositories (92% of all repositories)
violated at least one of the mined patterns. These numbers
show that the mined patterns did not represent only veryrare events in our dataset.
As discussed in § 5.1, we used cross-validation for model
selection. Now we report the cross-validation results forour selected model (support vector machine). We randomly
took 80% of labeled samples for training the model and thentested the trained model on the remaining 20% samples. We
repeated this for 30 times. The average accuracy, precision,
recall, and F1 score were 81%, 81%, 81%, and 80%, respectively.

Distribution of naming issues per pattern type. In ourevaluation, around 29% of the reports came from consistency
name patterns, while 81% of the reports were from confusing word patterns. The sum is above 100% because 10% of
the reports were detected by both patterns. To obtain moredetailed statistics on the precision of Namer per pattern
type, we ran Namer on all of the repositories in our dataset,
randomly selected 100 new reports for each of the two namepattern types, and manually inspected the selected reports.
The inspection results are shown in Table 4. We can see thatthe confusing word name pattern tended to recover more semantic defects, while the consistency name pattern produced
fewer false positives.

Learning to Find Naming Issues with Big Code and Small Supervision

Table 5. Precision of Namer and baselines on 300 randomly
selected violations from the Java dataset. "C" stands for thedefect classifier and "A" stands for the static analyses.
Baseline

Report

Namer
w/o C
w/o A
w/o C & A

97
300
138
300

5.3

Semantic Code quality
False
Precision
defectissue positive
2
2
0
0

64
90
66
87

31
208
72
213

68%
31%
48%
29%

PLDI ’21, June 20–25, 2021, Virtual, Canada

Table 6. Examples of reports by Namer for Java.
Index

Reported statement

12
3

e.getStackTrace();for(double i = 1; i < chainlength; i++) {} catch (Throwable e) {4
5
6

this.publicKey = publickKey;
context.startActivity(i);progDialog.dismiss();

Semantic defects

Precision of Namer. The results of Namer on the 300
selected violations from the Java dataset are shown in thefirst row of Table 5. The precision was 68%, similar to the precision of Namer for the Python dataset (70%), showing thatthe framework is generic and can precisely detect naming
issues for different types of languages.
Necessity of classifier and analyses. The last three rowsof Table 5 show the results when the classification module,
the analyses module, and both of them, were excluded from
Namer, respectively. Namer outperformed all three baselines
in precision by a large margin, reassuring the importance ofthe static analyses and the classifier.
Examples of reports found by Namer. We present examples of reports found by Namer for Java in Table 6:• Example 1 calls getStackTrace, which returns the stack
trace information, with receiver e (a thrown exception).
However, the return value is not assigned to any variable,
which makes the call redundant. As a result, the user might
miss important information about the exception. Namer
suggested to fix it by calling printStackTrace to print thestack trace to the standard error stream.
• Example 2 uses type double for loop index variable i, which
may cause unexpected program behavior due to floatingpoint operations. Namer suggested to use type int.
• Example 3 catches a Throwable variable e. Exception andError are both subclasses of Throwable. Catching Throwable
includes catching Error. However, the official documentation [2] suggests that Error should not be caught. Namercorrectly suggested to catch Exception instead of Error.
• Example 4–6 show cases where Namer suggested more
descriptive variable names than the original names. In Example 4, Namer detected and fixed a typo where public is
misspelled into publick. Example 5 is an Android API callfor creating a new activity. The argument of the function
startActivity is of type Intent and thus Namer suggested

print
int
Exception

Code quality issues
publicintentprogress

False positives

Results for Java

Next, we evaluate Namer on the Java dataset. The procedures
and experiments for the Java dataset were performed in thesame way as those for the Python dataset.

Suggested fix

7
8

final StringWriter outputWriter = ...;
ConektaObject resource = new ConektaObject();stringJson

to use the name intent instead of i. Example 6 dismisses
an Android dialog represented by the variable progDialog.
Namer suggested to replace prog with progress to indicatethat the dialog is a progress dialog.
• Examples 7 and 8 are two false positives. In Example 7,
Namer suggested to change the name outputWriter tostringWriter to better fit the class name StringWriter.
However, according to program context, the original nameis better. In Example 8, Namer suggested to replace class
ConektaObject with JsonObject because JsonObject is used
more frequently in our dataset. However, ConektaObject
should be the correct class.
Statistics on pattern mining and classification. For
the Java dataset, 79, 417 name patterns were mined, which
resulted in over 1.8 million violations. Moreover, 11% (i.e.,453, 450) of all the source files and 77% (i.e., 25, 496) of all therepositories had at least one violation. Similar to Python, themined patterns had a high coverage for the Java dataset. In
the cross-validation, the average accuracy, precision, recall,and F1 score were 90%, 90%, 90%, and 89%, respectively.
Distribution of naming issues per pattern type. For
the Java dataset, around 14.5% of the reports came from consistency name patterns and 91.7% of the reports were from
confusing word patterns. The sum is above 100%, because
6.2% of the reports were detected by both patterns. To obtain
more detailed precision statistics, we ran Namer on all therepositories in our evaluation and obtained 100 new reportsof each of the two bug pattern types. Then, we performeda manual inspection. For the consistency pattern, 0, 76 and
24 reports were classified as semantic defects, code quality
issues and false positives, respectively. For the confusing
word pattern, the number of semantic defects, code quality
issues and false positives was 3, 36 and 61, respectively.

PLDI ’21, June 20–25, 2021, Virtual, Canada

Jingxuan He, Cheng-Chun Lee, Veselin Raychev, and Martin Vechev

Table 7. Code quality issues selected for our user study described in § 5.4.
Issue category

Original code snippet

Detected issue & suggested fix

Inconsistent name

if docstring is not None :self . help = docstring

Minor Issue

def fullpath_set ( self , value ):self . _fullpath = value

Rename value to a more
descriptive name like fullpath

Confusing namedef __init__ ( self , song , ** kwargs ):self . _defaults = CODED_DEFAULTS
self . _factory = songself . set (** kwargs )Change some name to avoid
code like self._factory = songTypoself . port = por

Indescriptive namedef reset ( self , * e ):self . _autostep = 0

Rename help to docstring

Rename por to port
Rename e to a more descriptive name

Table 8. User study results of seven professional developers
on whether to accept code quality issues reported by Namer.

Table 9. Feature weights of the learned classifier.
FeatureNot
accepted

Accepted
with IDE
pluginAccepted
with pull
requestWouldeven fixmanually

Confusing nameIndescriptive name
Inconsistent name
Minor issue

0
0
2
2

3
3
0
4

2
2
4
0

2
2
1
1

Typo

1

2

1

3

Issue category

5.4

User Study on Severity of Code Quality Issues

To gain more insights into the severity of code quality issues
found by Namer, we performed a small user study showing
5 Python reports to professional developers from a software
company with no mandatory naming style guides. The 5
reports, shown in Table 7, were chosen by randomly picking one sample from each category in the breakdown ofTable 4 and were shuffled. The participants were not awareof Namer and were not told that the reports were provided
by an automatic tool. They were asked if and at what conditions they would accept the changes in the reports as the
project maintainers. The conditions include (i) at coding timewith an automatic IDE plugin, (ii) after coding time with anautomatic pull request, and (iii) the developer is willing to
fix the issue manually once seeing it.
The study received 7 responses which are shown in Table 8.
The results demonstrate that the developers found the issues
relevant. Only in 5 cases, an issue was not accepted and in
9 cases, the developers were even willing to fix the issues
themselves. For most cases, developers accepted a report
only if an automatic tool like Namer, either in the form of
an IDE plugin or pull requests, locates the issue and suggests
a fix for them.

Identical statement
Satisfaction count
Violation count

5.5

File level

Repo level

Entire dataset

0.6345
1.86-1.121-2.8540.468-1.0655-0.7305
1.5565

Understanding Classifier Decision Making

Seeing the effectiveness of Namer, one natural question to
ask is why the classifier performed well and based on what
the classifier made decisions. Since we used a linear modelon standardized features, the weights of the learned classifier
are useful in measuring how much each individual feature
contributes to the final decision. In Table 9, we summarize
the feature weights averaged from the learned classifiers for
Python and Java. The three rows show the weights of feature
2-3, 4-6, and 7-9, respectively.
We can see that the absolute values of all weights were
non-negligible, meaning that the classifier makes decisions
based on various features. Moreover, the classifier learned
interesting behaviors non-obvious to a human: the contribution of the same type of feature can be completely opposite
over different levels. For example, for the violation countfeature, the weights over file and repository level were negative but over the entire dataset, the weight was positive. Thisphenomenon demonstrates that jointly considering both local and global features is another key reason for the classifier
to precisely distinguish between true issues and false positives. The above observations are in contrast to previous
works [34, 38, 41] which involves only one or few weakpredictors, and can give insights for future bug finding tools.
5.6

Comparison with Deep Learning Approaches

We now compare Namer with two state-of-the-art deeplearning based approaches for finding name based issues,

Learning to Find Naming Issues with Big Code and Small Supervision

Table 10. Precision comparision of GGNN, Great and Namer
on 134 randomly selected reports for Python.
SystemSemantic
defects

Code quality
issuesFalse
positives

GGNN
Great
Namer

1
2
5

20
9
89

113
123
40

PLDI ’21, June 20–25, 2021, Virtual, Canada

Table 11. Precision comparision of GGNN, Great and Nameron 97 randomly selected reports for Java.

Precision

SystemSemantic
defects

Code quality
issuesFalse
positives

16%
8%
70%

GGNN
Great
Namer

2
2
2

7
3
64

88
92
31

GGNN [9] and Great [28]. Both works leverage deep neuralnetworks over program graphs to detect misuses of variablenames and reported high accuracy on testing datasets withsynthetic bugs. However, the high testing accuracy does notnecessarily reflect their capability of finding real world issues. This is because the distribution of true issues is notnecessarily the same as the distribution of synthetic issues
used to train and test the network models. In fact, both works
presented only limited evaluation results on finding real problems in code. The authors of GGNN presented case studies
on bugs but did not include quantitative measures. Great’s
precision dropped to 29.4% on a dataset of only hundreds of
buggy and non-buggy programs that the authors collectedby comparing Github commits. Therefore, the practical effectiveness of GGNN and Great on finding real world issues
is still unknown. Through our evaluation, we aim to understand the practical effectiveness of these works and compare
their precision with Namer.
Training and measuring accuracy. We re-implemented
the pipeline of GGNN and Great based on the machine learning modules open sourced by the authors [4, 5]. To obtaintraining program graphs, we followed the original works[9, 28] to introduce synthetic changes to the programs in our
Python and Java datasets. Note that it is impossible to train
GGNN and Great with real issues as we did with Namer.
This is because deep networks are data hungry and a datasetwith a sufficient amount of real issues does not exist and is
hard to obtain with either manual labor or automation. The
training processes took around 70h to 130h.
After training, we measured the accuracy of GGNN andGreat to ensure that each individual model can achieve highaccuracy as reported in the original papers [9, 28]. We did not
aim to compare the accuracy across those models. For eachmodel, we sampled a subset of programs from our dataset, introduced synthetic changes to them, and constructed 10, 000
new program graphs. The new graphs were not used in training though the original programs could be. Then, we fed thegraphs into the trained models and obtained the accuracy. For
GGNN, the accuracy was 71% (resp., 83%) for Python (resp.,
Java). For Great, the classification accuracy, localization accuracy and repair accuracy for Python (resp., Java) were 91%,
83%, and 79% (resp., 91%, 82%, and 81%), respectively.

Precision
9%
5%
68%

From these results on accuracy, we could reproduce thefact that GGNN and Great result in high accuracy on thedataset of synthetic code changes. Further, these numbers
closely match the results in the original papers [9, 28].
Precision of detecting issues. Next, we evaluate GGNN
and Great on finding real world issues. To this end, we tested
GGNN and Great on our dataset without any synthetically introduced changes. Same as Namer, we inspected a randomly
selected set of issue reports for GGNN and Great. All truevariable misuses were classified as semantic issues as theyusually cause unexpected program behaviors. We found thatsome reports were not variable misuses but pointed to otherkinds of naming issues, e.g., unused variables. We included
them as true positives and classified them into semantic
issues and code quality issues based on our criterion in § 5.1.
A factor that affects the precision and recall of a bug finding tool is the confidence level above which a report is made.
For GGNN and Great, we could not determine the total number of true issues that can be theoretically reported by therespective models in comparison to the total number of issues discoverable by Namer, but based on a sample of thefrequency of the issues in code, we estimated that a namemisuse predictor should be applicable to at least one fifthof the naming problems. To compensate for this effect, we
tuned the confidence levels so that both GGNN and Great
reported around 5× fewer issues than Namer.
The results on randomly inspected reports for Python are
shown in Table 10. Both GGNN and Great found very few
true issues, resulting in the reported low precision: GGNN
achieved 16% precision with 1 semantic defect and 20 code
quality issues; Great achieved 8% precision with 2 semantic
defects and 9 code quality issues. This low detection capabilities of the networks do not mean that there are no true issuesof the specific kind, it just shows that the probability distribution of issues encoded in the predictor did not correlatewith that of real naming issues. As shown in Table 11, thenetworks also achieved low precision for Java (9% for GGNN
and 5% for Great). We tried to increase the confidence levels
of GGNN and Great to the maximum level, i.e., inspecting
the most confident reports, as done by [9, 39]. This enabled
GGNN and Great to find more semantic issues and increased
their precision to ∼40%. However, with such high confidencePLDI ’21, June 20–25, 2021, Virtual, Canada

levels, the recall became nearly zero, which is not desired inpractice, and the precision was still non-satisfactory.
Compared to GGNN and Great, Namer has a much higher
precision of ∼70%, despite returning 5× more reports. Another important aspect is that the naming rules mined by
Namer and the decision making process of the classifier
are highly interpretable, while neural code models are noninterpretable and are vulnerable to adversarial attacks [16].
As a result, developers can inspect and better understandthe issues reported by Namer. Therefore, we conclude that
Namer is practically more effective than GGNN and Great.

6

Related WorkIn this section, we discuss works mostly related to ours.
6.1

Naming issue detection and repair

Existing techniques on naming issue detection can be categorized into two kinds: rule-based and learning-based. Namer
can be seen as a combination of the two types of techniques.
Rule-based approaches. The works of [30, 34, 38] leverage rule-based anomaly detection for finding naming issues.
The detection solely depends on extracted rules [30] or similarity measures [34, 38], while Namer’s decision process
considers more factors and features. The work of [41] uses
a graph matching algorithm to detect argument selection
defects in an industry level codebase. The above approaches
only focus on a specific type of naming issue (wrong usagesof function arguments) and a specific language (Java). On
the contrary, Namer can detect more kinds of naming issues
and targets both Java and Python.
Learning-based approaches. A number of recent works
propose machine learning techniques for name-based bug
finding. Apart from GGNN [28] and Great [28] discussed in§ 5.6, GINN [47] is another graph neural network for codethat leverages a hierarchy of intervals over the control flow
graph for better message passing. For detecting variable misuses, GINN achieves higher accuracy than GGNN and Great,but suffers from the same limitation: synthetic issues areused for training. DeepBugs [39] learns embeddings for codesnippets to detect three types of naming bugs for Javascript:
swapped arguments, wrong binary operator, and wrong binary operand. DeepBugs achieves high precision on the mostconfident reports (i.e., low number of reports). It is unclear
how effective DeepBugs is in a realistic scenario where ahigher number of reports is desired.
Instead of detecting bugs, other works learn probabilistic
models for predicting or suggesting identifier names. Probabilistic graphical models are employed in [15, 27, 40] for predicting identifier names for minified Javascript program, obfuscated Android applications and stripped binaries, respectively. Naturalize [8] suggests identifier names by learning
statistical language models representing coding conventions.

Jingxuan He, Cheng-Chun Lee, Veselin Raychev, and Martin Vechev

Convolutional attention networks are used in [10] for codesummarization, i.e., method name suggestion. Other notableworks [11, 13] use paths between AST nodes as input features to a neural network trained to summarize code andpredict descriptive method names.
6.2

Machine Learning and Program Reasoning

In the following, we discuss research at the intersection ofmachine learning and program reasoning.
Static analysis. Bayesian optimization [36] and reinforcement learning [29, 43] have been used for balancing thetradeoff between precision and performance for numericalprogram analysis. API specifications are a key ingredient inmodern static analysis. Atlas [14] leverages active learning
to obtain points-to specifications. Uspec [22] learns API aliasing specifications with unsupervised learning. Seldon [20]infers taint specifications by solving linear optimization. Statistical methods have been used to rank and classify alertsfrom static analyzers [25, 31].
Testing. Machine learning has been adopted to improve
test input generation. Skyfire [46] learns from an input corpus and input grammar for generating well-distributed seed
inputs for fuzzing. AFLFast [17] learns a markov chain formodeling program branching behavior to guide input generation. The work of [42] learns feed-forward networks from already generated fuzzing inputs to guide future mutation. ILF[26] learns gate recurrent units from input traces generated
by symbolic execution. The work of [23] learns recurrent
neural networks from an existing test corpus.

7

Conclusion

We proposed a novel approach for finding and fixing naming
issues. Our method combines interpretable name patterns
mined from Big Code to represent different kinds of naming
idioms together with a machine learning classifier that filters
false positives, trained on expressive features. Importantly,
making use of Big Code does not require labeling the programs (which would be practically infeasible), while we only
require a small amount of labeled data to train the classifier.
We implemented our approach in a system called Namer
which supports both Python and Java, and evaluated it extensively on a massive dataset of open source repositories.
Our experimental results showed that Namer is able to findreal world naming issues and suggest correct fixes with highprecision (∼70%). We also showed that Namer is practicallymore effective than state-of-the-art deep neural network
approaches (that use synthetic data).
We believe this work presents new insights in building apractical bug finding tool with machine learning. It also reassures the importance of program abstraction and analyses
in finding defects and other issues in real world code.

Learning to Find Naming Issues with Big Code and Small Supervision

References[1] 2020. American fuzzy lop. https://lcamtuf.coredump.cx/afl/[2] 2020. Error (Java SE 14 & JDK 14). https://docs.oracle.com/en/java/
javase/14/docs/api/java.base/java/lang/Error.html[3] 2020. GitHub. https://github.com[4] 2020. ICLR20-Great. https://github.com/VHellendoorn/ICLR20-Great[5] 2020. tf-gnn-samples. https://github.com/microsoft/tf-gnn-samples[6] 2020. unittest — Unit testing framework. https://docs.python.org/3/
library/unittest.html#unittest.TestCase.assertTrue[7] Miltiadis Allamanis, Earl T. Barr, Christian Bird, and Charles Sutton.
2015. Suggesting accurate method and class names. In FSE 2015. https://doi.org/10.1145/2786805.2786849[8] Miltiadis Allamanis, Earl T. Barr, Christian Bird, and Charles A. Sutton.
2014. Learning Natural Coding Conventions. In FSE 2014. https://doi.org/10.1145/2635868.2635883[9] Miltiadis Allamanis, Marc Brockschmidt, and Mahmoud Khademi.
2018. Learning to Represent Programs with Graphs. In ICLR 2018.
https://openreview.net/forum?id=BJOFETxR[10] Miltiadis Allamanis, Hao Peng, and Charles A. Sutton. 2016. A Convolutional Attention Network for Extreme Summarization of Source Code.
In ICML 2016. http://proceedings.mlr.press/v48/allamanis16.html[11] Uri Alon, Shaked Brody, Omer Levy, and Eran Yahav. 2019. code2seq:
Generating Sequences from Structured Representations of Code. In
ICLR 2019. https://openreview.net/forum?id=H1gKYo09tX[12] Uri Alon, Meital Zilberstein, Omer Levy, and Eran Yahav. 2018. A
General Path-Based Representation for Predicting Program Properties.
In PLDI 2018. https://doi.org/10.1145/3192366.3192412[13] Uri Alon, Meital Zilberstein, Omer Levy, and Eran Yahav. 2019.
code2vec: Learning Distributed Representations of Code. Proc. ACM
Program. Lang. 3, POPL (2019), 40:1–40:29. https://doi.org/10.1145/
3290353[14] Osbert Bastani, Rahul Sharma, Alex Aiken, and Percy Liang. 2018.
Active Learning of Points-to Specifications. In PLDI 2018. https://doi.
org/10.1145/3192366.3192383[15] Benjamin Bichsel, Veselin Raychev, Petar Tsankov, and Martin Vechev.
2016. Statistical Deobfuscation of Android Applications. In CCS 2016.
https://doi.org/10.1145/2976749.2978422[16] Pavol Bielik and Martin Vechev. 2020. Adversarial Robustness forCode. In ICML. http://proceedings.mlr.press/v119/bielik20a.html[17] Marcel Böhme, Van-Thuan Pham, and Abhik Roychoudhury. 2016.
Coverage-based Greybox Fuzzing as Markov Chain. In CCS 2016. https://doi.org/10.1145/2976749.2978428[18] Simon Butler, Michel Wermelinger, Yijun Yu, and Helen Sharp. 2010.
Exploring the Influence of Identifier Names on Code Quality: An
Empirical Study. In CSMR 2010. https://doi.org/10.1109/CSMR.2010.27[19] Cristian Cadar, Daniel Dunbar, and Dawson R. Engler. 2008. KLEE:
Unassisted and Automatic Generation of High-Coverage Tests forComplex Systems Programs. In OSDI 2008. http://www.usenix.org/
events/osdi08/tech/full_papers/cadar/cadar.pdf[20] Victor Chibotaru, Benjamin Bichsel, Veselin Raychev, and Martin
Vechev. 2019. Scalable Taint Specification Inference with Big Code. In
PLDI 2019. https://doi.org/10.1145/3314221.3314648[21] Patrick Cousot and Radhia Cousot. 1977. Abstract Interpretation: A
Unified Lattice Model for Static Analysis of Programs by Construction
or Approximation of Fixpoints. In POPL 1977. https://doi.org/10.1145/
512950.512973[22] Jan Eberhardt, Samuel Steffen, Veselin Raychev, and Martin Vechev.
2019. Unsupervised Learning of API Aliasing Specifications. In PLDI
2019. https://doi.org/10.1145/3314221.3314640[23] Patrice Godefroid, Hila Peleg, and Rishabh Singh. 2017. Learn&Fuzz:
Machine Learning for Input Fuzzing. In ASE 2017. https://doi.org/10.
1109/ASE.2017.8115618[24] Jiawei Han, Jian Pei, and Yiwen Yin. 2000. Mining Frequent Patterns
without Candidate Generation. In SIGMOD 2000. https://doi.org/10.

PLDI ’21, June 20–25, 2021, Virtual, Canada
1145/342009.335372[25] Quinn Hanam, Lin Tan, Reid Holmes, and Patrick Lam. 2014. Finding
patterns in static analysis alerts: improving actionable alert ranking.
In MSR 2014. https://doi.org/10.1145/2597073.2597100[26] Jingxuan He, Mislav Balunovic, Nodar Ambroladze, Petar Tsankov,
and Martin Vechev. 2019. Learning to Fuzz from Symbolic Execution
with Application to Smart Contracts. In CCS 2019. https://doi.org/10.
1145/3319535.3363230[27] Jingxuan He, Pesho Ivanov, Petar Tsankov, Veselin Raychev, and Martin Vechev. 2018. Debin: Predicting Debug Information in Stripped
Binaries. In CCS 2018. https://doi.org/10.1145/3243734.3243866[28] Vincent J. Hellendoorn, Charles Sutton, Rishabh Singh, Petros Maniatis,
and David Bieber. 2020. Global Relational Models of Source Code.
In ICLR 2020. OpenReview.net. https://openreview.net/forum?id=
B1lnbRNtwr[29] Kihong Heo, Hakjoo Oh, and Hongseok Yang. 2019. Resource-aware
Program Analysis via Online Abstraction Coarsening. In ICSE 2019.
https://doi.org/10.1109/ICSE.2019.00027[30] Einar W. Høst and Bjarte M. Østvold. 2009. Debugging Method Names.
In ECOOP 2009. https://doi.org/10.1007/978-3-642-03013-0_14[31] Ted Kremenek and Dawson R. Engler. 2003. Z-Ranking: using statisticalanalysis to counter the impact of static analysis approximations. In
SAS 2003. https://doi.org/10.1007/3-540-44898-5_16[32] Carson Kai-Sang Leung, Laks V. S. Lakshmanan, and Raymond T.
Ng. 2002. Exploiting Succinct Constraints using FP-trees. SIGKDD
Explorations 4, 1 (2002), 40–49. https://doi.org/10.1145/568574.568581[33] Yi Li, Shaohua Wang, Tien N. Nguyen, and Son Van Nguyen. 2019.
Improving Bug Detection via Context-Based Code Representation
Learning and Attention-Based Neural Networks. Proc. ACM Program.
Lang. 3, OOPSLA (2019), 162:1–162:30. https://doi.org/10.1145/3360588[34] Hui Liu, Qiurong Liu, Cristian-Alexandru Staicu, Michael Pradel, andYue Luo. 2016. Nomen est omen: Exploring and Exploiting Similarities
between Argument and Parameter Names. In ICSE 2016. https://doi.
org/10.1145/2884781.2884841[35] Cristina V. Lopes, Petr Maj, Pedro Martins, Vaibhav Saini, Di Yang,
Jakub Zitny, Hitesh Sajnani, and Jan Vitek. 2017. DéjàVu: a Map of
Code Duplicates on GitHub. Proc. ACM Program. Lang. 1, OOPSLA(2017), 84:1–84:28. https://doi.org/10.1145/3133908[36] Hakjoo Oh, Hongseok Yang, and Kwangkeun Yi. 2015. Learning a
Strategy for Adapting a Program Analysis via Bayesian Optimisation.
In OOPSLA 2015. https://doi.org/10.1145/2814270.2814309[37] Rumen Paletov, Petar Tsankov, Veselin Raychev, and Martin Vechev.
2018. Inferring crypto API rules from code changes. In PLDI 2018.
https://doi.org/10.1145/3192366.3192403[38] Michael Pradel and Thomas R. Gross. 2011. Detecting Anomalies in
the Order of Equally-typed Method Arguments. In ISSTA 2011. https://doi.org/10.1145/2001420.2001448[39] Michael Pradel and Koushik Sen. 2018. DeepBugs: a Learning Approach to Name-based Bug Detection. Proc. ACM Program. Lang. 2,
OOPSLA (2018), 147:1–147:25. https://doi.org/10.1145/3276517[40] Veselin Raychev, Martin Vechev, and Andreas Krause. 2015. Predicting
Program Properties from "Big Code". In POPL 2015. https://doi.org/10.
1145/2676726.2677009[41] Andrew Rice, Edward Aftandilian, Ciera Jaspan, Emily Johnston,Michael Pradel, and Yulissa Arroyo-Paredes. 2017. Detecting Argument Selection Defects. Proc. ACM Program. Lang. 1, OOPSLA (2017),
104:1–104:22. https://doi.org/10.1145/3133928[42] Dongdong She, Kexin Pei, Dave Epstein, Junfeng Yang, Baishakhi Ray,and Suman Jana. 2019. NEUZZ: Efficient Fuzzing with Neural Program
Smoothing. In S&P 2019. https://doi.org/10.1109/SP.2019.00052[43] Gagandeep Singh, Markus Püschel, and Martin Vechev. 2018. Fast
Numerical Program Analysis with Reinforcement Learning. In CAV
2018. https://doi.org/10.1007/978-3-319-96145-3_12

PLDI ’21, June 20–25, 2021, Virtual, Canada[44] Yannis Smaragdakis and George Balatsouras. 2015. Pointer Analysis.
Found. Trends Program. Lang. 2, 1 (2015), 1–69. https://doi.org/10.1561/
2500000014[45] Marko Vasic, Aditya Kanade, Petros Maniatis, David Bieber, andRishabh Singh. 2019. Neural Program Repair by Jointly Learning
to Localize and Repair. In ICLR 2019. https://openreview.net/forum?
id=ByloJ20qtm

Jingxuan He, Cheng-Chun Lee, Veselin Raychev, and Martin Vechev[46] Junjie Wang, Bihuan Chen, Lei Wei, and Yang Liu. 2017. Skyfire: DataDriven Seed Generation for Fuzzing. In S&P 2017. https://doi.org/10.
1109/SP.2017.23[47] Yu Wang, Ke Wang, Fengjuan Gao, and Linzhang Wang. 2020. Learning
semantic program embeddings with graph interval neural network.
Proc. ACM Program. Lang. 4, OOPSLA (2020), 137:1–137:27. https://doi.org/10.1145/3428205