Learning to Mine Aligned Code and Natural
Language Pairs from Stack Overflow

Pengcheng Yin* Bowen Deng* Edgar Chen Bogdan Vasilescu Graham Neubig
Carnegie Mellon University

Background• When Natural Language Processing (NLP) meets Software Engineering…
Code Summarization1

1

SELECT Max(marks) FROM records
WHERE marks <(SELECT Max(marks) FROMrecords);

Get the second largest value of acolumn

Code Retrieval22

Code Generation3

3

load csv file to pandas

sort my_list in descending orderpandas.read_csv(’example.csv’)sort(my_list,reverse=True)• These tasks, mostly powered by data-driven models, heavily rely on parallel training (andevaluation) corpora of source code and natural language in high quality and large amount[1] Iyer et al., ACL ’16; Allamanis et al., ICML ‘16[2] Zhang et al., FSE ’16; Gu et al., ICSE ‘18[3] Raghothaman et al., ICSE ’16; Yin et al., ACL ‘17, ACL ‘18

Collecting Intent/Snippet Pairs from SO• Such data-driven models require parallel data of natural language intents and sourcecode snippets and in high volume and high quality– Intent natural language description of what a programmer would like to do– Snippet a piece of source code that implements the intent
Intent get the maximum value of a column
Snippet SELECT MAX(marks) from records

Intent read a csv file into pandas
Snippet pandas.read_csv(‘example.csv’)Intent sort my_list in descending order
Snippet sort(my_list, reverse=True)

Can we collect such data from?
Heuristic approaches [Wong et al., 2013;
Iyer et al., 2016]?
- Select all code blocks- Select all code blocks in accepted answers...

Are these Heuristic Approaches Good Enough?
IntentHowever, acquiring such datafrom Stack Overflow posts may
not be that straight-forward ….

ContextualInformation
Actual Snippet
Auxiliary info.
Irrelevant Code

1. Contextual Information package
import statements, variable
definition
2. Auxiliary information return values,
example outputs
3. Irrelevant Code
4. Counter-examples

Heuristic approaches?
- Select all code blocks- Select all code blocks in acceptedanswers

Our Solution• CONALA, a system to collect parallel data of source code snippet and natural
language intents from Stack Overflow– data-driven: learn patterns of “good” and “bad” intent/snippet pairs from data
(using neural networks)– language-agnostic: applicable to different programming languages (e.g., Pythonand Java)– Scalable: capable of applying to full-scale Stack Overflow data (collected ~600K
intent/snippet pairs for Python)

Project Website: conala-corpus.github.io

System Architecture• Enumerate all possible candidate intent/snippet pairs in a Stack Overflow page• Learn a classifier to rank each candidate intent/snippet pair

Question (i.e. intent)removing duplicates in lists
An Answer Code Blockt = [1, 2, 3]list(set(t))
s = [1, 2, 3]list(set(t) – set(s))

Input SO question(“how-to” questions)t = [1, 2, 3]list(set(t))

p=0.6list(set(t))

list(set(t) – set(s))

p=0.2

t = [1, 2, 3]list(set(t))

p=0.06t = [1, 2, 3]

p=0.04s = [1, 2, 3]list(set(t) – set(s))

p=0.03list(set(t) – set(s))
t = [1, 2, 3]list(set(t))
s = [1, 2, 3]list(set(t) – set(s))

Generated by enumerating
contiguous sequence of codein answer code blocks

...

...
Candidate Snippets

Feature-basedClassifier

Ranked Snippets

Features• Purpose measure the probability (plausibility) of each intent/snippet candidate• Two types of features language independent, highly-indicative= removing duplicates in lists
P{correct | Intent}
Snippet = list(set(t))
Inspired by heuristic approaches, does not use intent information
StructuralFeatures

Code Shape features: FULLBLOCK CONTAINSIMPORT STARTSWITHASSIGNMENT …
Answer quality features: ISACCEPTEDANSWER POSTRANK …

Powered byUse state-of-the-art neural networks to estimate the (semantic)
correspondence between and intent and snippet
Correspondence
Features

Score{ removing duplicates in lists , list(set(t))}

Neural Correspondence Model between Code and Intent• Neural sequence-to-sequence networks for translation probability between
intent and snippet! "#$%#$ &#"!!%$ = 0.14

ﬁnd

Encoding Recurrent Neural
Network (RNN) encodes theinformation of the code

my_list.

index(indexof"bar"


Decoding RNN generates
the intent using theencoded source code"bar")

Neural Net for Snippet 7! IntentNo Need to Use Curated Training Data
using large amounts of question titles andcode blocks from accepted answers to train
the model

What Left to be Learned?
• Structural Features shown by existing works as indicative• Correspondence Features already pre-trained on massive, readily available
data on Stack Overflow (questions and code blocks)!
• We just need small amount of manually annotated intent/snippet pairs to tune
the ~20 weights in the classifier
Gold Annotations
527 examples
142 questions
330 examples
100 questions

Cross-Validation ResultsPrecision

1.00
0.75
0.50

Structural and correspondence
features are complimentary

Full System

Code Blocks in
Accepted Answers

Use All Code Blocks
Structural Features

0.25
0.00
0.0

Structural Features
Correspondence Features

0.20.4

0.6
Recall0.8

1.0

Must we Annotate each Language?
• The classifier requires small set of gold-standard annotations to train on• When apply our system to a new language, can we use the existing modeltrained on another (old) language?

Precision

1.00
Python Model

0.75
0.50

Java Model

0.25

Python Model w/ Structural Features Only

0.00
0.0

0.2

0.4

0.60.8Recall
Apply the model trained on Java data to mine Python intent/snippet data

1.0

Dataset Collection• Apply the system to Python questions on Stack Overflow, collecting ~600K pairs• ~2500 (and counting) high-quality annotated intent/code snippet pairs• Rewritten Intents manually annotated, revised intents to reflect the full meaningof the code
– Add free variable names, arguments to the intent– Useful for fine-grained language to code tasks like code generation
Crowdsourced

Intent copying one file's contents to another
Rewritten Intent copy the content of file 'file.txt' to file 'file2.txt'
Code Snippet shutil.copy('file.txt','file2.txt')
An example from the annotation dataset

Examples• Examples covers a wide variety
of use cases– Built-in data type operation– I/O and string operation– Third-party library usage• Examples are highly expressive
and compositional!
– Pose challenges to existingcode/NL models

Intent dict how to create key or append an element to key
Rewritten Intent Create a key `key` if it does not exist in dict `dic` and append
element `value` to value
Code Snippet dic.setdefault(key, []).append(value)
Intent How do I check if all elements in a list are the same
Rewritten Intent check if all elements in list `mylist` are the same
Code Snippet len(set(mylist)) == 1
Intent Iterate through words of a file in Python
Rewritten Intent check if all elements in list `mylist` are the same
Code Snippet words = open('myfile').read().split()
Intent Delete Column in Pandas based on Condition
Rewritten Intent delete all columns in DataFrame `df` that do not hold a non-zerovalue in its records
Code Snippet df.loc[:, ((df != 0).any(axis=0))]

Check our Dataset atconala-corpus.github.io

Annotate Gold-standard Dataset• Our system needs a small set of gold-standard intent/snippet data to learn theparameters of the classifier powered by high-level features• Annotate top-ranked how-to questions on Stack Overflow for each languageGold Annotations
527 examples
142 questions
330 examples
100 questions

System Deployment• We deploy our system on the top-50K Python-tagged questions on Stack
Overflow, and collected ~600K ranked intent/code-snippet pairs(50K)