NLP in Real World Settings:
Healthcare & Program Synthesis
Ndapa Nakashole

13 July, 2023

1

Asking Health Questions

For labor induction, what is the medication that is
FDA approved for cervical ripening?
ChatGPT’s answer (GPT-3.5): Misoprostol

2The FDA says ...

Correct answer: Dinoprostone( GPT-4 get is right)
3

Transformer knowledge storage mechanism▶ Where is the incorrect
belief stored?
⊙ Token embeddings?
⊙ Feedforward layers?
⊙ Attention layers?

4Transformer storage mechanism

Transformer
feed-forward
layersarekey-value m
Finding 1 : Feed-forward
layers in TransformerLMsemulatekey-value(Geva
et memories
al, 2021)Transformer feed-forward layers are key-value memories(Geva et al, 2021)

output (y)output (y)W2W2nonlinearity (𝞂)W1nonlinearity (𝞂)input (x)I have omitted bias terms, layer norm, residual connections.

W1

input (x)1

I have omitted bias terms, layer norm, residu[Geva et al.(2021)Geva, Schuster, Berant, and Levy]

5

Modifying the memory
Let's see what memories were selected
Zooming in on an FF layer
output (y)
FF

FF

FF

FF

FF

FF

FF

FF

Att

Att

Att

Att

Att

Att

Att

Att

FF

FF

FF

FF

FF

FF

FF

FF

nonlinearity (𝞂)

Att

Att

Att

Att

Att

Att

Att

Att

W1

Eiffel Tower is located in the city of

1

Figure from: K. Guu

W2

input (x)6Modifying the memory
Modifying the memory
Intuition: modify columns of W2 tochange model's behavior.

output (y)W2Subtract word vector for Paris, add
word vector for Rome?
(Dai et al, 2021)

selector
nonlinearity (𝞂)W1input (x)Meng et al, 2022: apply a rank-1 update.
●●●W2 ← W2 + uvT (u and v are vectors)
Maximize probability of outputting Rome when we
see "Eiffel Tower" selector.
Minimize change in behavior of W2 on other inputs.

Editing approches are still in their infancy.
1

Figure from: K. Guu

7

Desired from the Transformer▶ Attribution and interpretability⊙ Trace a model’s knowledge back to a particular document or
training example▶ Modular knowledge editing⊙ Update the knowledge the model has access to without
breaking its behavior▶ Efficient scaling⊙ Increase the model’s memory size by 10x without paying 10x
more compute.

8

Memory-Augmented Models

9The Memory Extender▶ Allow LM to interact
with external memory(search engine, DB,etc):▶ Thus supporting:
Editing of knowledge,
Attribution, Efficientscaling

Many approaches have been proposed:▶ Memory Networks▶ REALM▶ DPR▶ LaMDA▶ WebGPT▶ ...
▶ RETRO

10

Retrieval-Enhanced Transformer (RETRO), Borgeaud et al.
(ICML 2022)
Improving language models by retrievingfrom trillions of tokens
Sebastian Borgeaud† , Arthur Mensch† , Jordan Hoffmann† , Trevor Cai, Eliza Rutherford, Katie Millican,
George van den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, Diego de Las Casas,
Aurelia Guy, Jacob Menick, Roman Ring, Tom Hennigan, Saffron Huang, Loren Maggiore, Chris Jones,
Albin Cassirer, Andy Brock, Michela Paganini, Geoffrey Irving, Oriol Vinyals, Simon Osindero,
Karen Simonyan, Jack W. Rae‡ , Erich Elsen‡ and Laurent Sifre†,‡All authors from DeepMind, † Equal contributions, ‡ Equal senior authorship

We enhance auto-regressive language models by conditioning on document chunks retrieved from a[cs.CL] 7 Feb 2022

2

large corpus,
on local similarity
with database
preceding tokens. →Withcomparable
a 2 trillion token database, our▶ Memory:a based
2 trillion
token
Retrieval-Enhanced Transformer (R) obtains comparable performance to GPT-3 and Jurassic-1
performance
tousing
GPT-3
others,but with
25×
fewer
on the Pile, despite25⇥ fewerandparameters.
After fine-tuning,Rperformancetranslates todownstream knowledge-intensive tasks such as question answering. R
combines a frozen B
parameters
retriever, a differentiable encoder and a chunked cross-attention mechanism to predict tokens based on

2an order of magnitude more data than what is typically consumed during training. We typically trainRfrom scratch, yet can also rapidly R
fit pre-trained transformers with retrieval and stillachieve good performance. Our work opens up new avenues for improving language models throughexplicit memory at unprecedented scale.

1. Introduction[Borgeaud et al.(2022)Borgeaud, Mensch, Hoffmann, Cai, Rutherford, Millican, van den Driessche, Lespiau,
11 D

RETRO Method

Improving language models by retri▶ Keys, values derived from afrozen BERT
▶ Training sequences are
divided into chunks andaugmented with theirk-nearest neighbor chunks
from the database.
▶ Encoder-decoder ,integrates
retrieval chunks using
cross-attention

Encoded neighbours

Neighbours

RetrievaldatasetTransformer
Encoder▶ Construct a key-value
database, where values
store raw chunks of text
tokens

BERT
BERT

E1
E2

Frozen kNN Retriever

AttendCondition
Input
tokens

K

V

C1

EMB

C2

ATTN

Q

CCA

FFW

READ

C3
X

RETRO block (x L)

Figure 2 | R
architecture. Left: simplified ver
into : = 3 chunks of size ; = 4. For each chunk, we ret
retrieval pathway is shown on top. Right: Details of th
maintained as neighbours of the first chunk only aff
from the second chunk.

a new methodology to evaluate language models whtraining set.
12

Design Considerations for Memory-Augmented Models▶ What goes o the memories?
⊙ Documents, database records, training examples, etc▶ How to use the retrieved memories?
⊙ "Text fusion", "label smearing"▶ How to retrieve memories?
⊙ Use an off-the-shelf search engine⊙ How to train your own memory retriever

13Case Study # 1: Healthcare QA

14

Memory models for Healthcare QA▶ Our memory representation: Question-Answer pairs (ACL
2021, Mrini et al)▶ QA pairs are concise, and also tend to express importation
information

15

Consumer Health Question (CHQ) Answering(CHQ) I am a new mother, blessed with a beautiful baby justa few weeks ago. While this journey has been an incredible one
filled with love and joy, it’s also been quite overwhelming attimes. I am concerned about postpartum depression (PPD) as
I might have it. I’m unsure about its exact causes and would
really appreciate if someone could provide more clarity on this.
(FAQ) What is the physiological basis for postpartum depression?

16

Approach
300
301
302
303
304

summarization pair. If it is not an entailment case,then we have: (3) , question F is not a summaryof question C if and only if question C does notentail question F (4). Therefore, to create an equivalent RQE pair labeled as not entailement, the RQE
CHQ is identical to the CHQ of the summarization
pair, and the RQE FAQ is randomly selected froma distinct question pair from the same dataset split.
Inversely, for the RQE dataset, we create equivalent summarization pairs. For each existing RQEpair, we consider two cases. If the RQE pair islabeled as entailment, we create an identical summarization pair. If the RQE pair is labeled as notentailment, then following (4), we create a summarization pair that is identical to a randomly selected
and distinct RQE pair labeled as entailment fromthe same dataset split.

Gradually SoftParameter-Sharing
Loss
Negative
Log-Likelihood
Loss

305
▶ Approach:
Key: FAQ; Value: FAQ
306
307
answers308

Decoder

Cross-Entropy
LossClassiﬁcation
Head

Decoder

309 CHQ, an answer is obtained via
▶ For
310
label
smearing
311▶ A312
multi-task learning approach for313
314
CHQs, summarize and match
315
perform
matching via question
316
317
entailment
3.3 Simultaneous Multi-Task Learning
318

319
Previous work on multi-task learning with sum▶ Encoder-decoder
320
321
322
323

marization and entailment (Pasunuru et al., 2017;
Guo et al., 2018) optimize for the objectives of thedifferent tasks by alternating between them. This
alternating multi-task training follows a ratio between the different tasks, that depends on the sizeof the dataset of each task (e.g. a ratio of 10:1
means training for 10 batches on one task, and thenfor 1 batch on the other task). In our approach, we
propose to optimize simultaneously for the objectives of both tasks. We do not use ratios, as we are
not alternating between objectives and the resulting

FAQ

CHQ; FAQ
SharedEncoder

CHQ
Question
Summarization
Generation Task

CHQ; FAQ
Recognizing Question
Entailment (RQE)
Classiﬁcation Task

Figure 2: Overview of the architecture of our proposed
gradually soft multi-task and data-augmented model.
The gradually thinning links between decoder layers
represent the loosening parameter-sharing constraint.

324
ROUGE
and human evaluation show effectiveness of approach &325
BioNLP
shared task system submission placedcompetitively
326
We propose
to optimize a single loss function
327
328
329

that combines objectives of both tasks. Our loss
function is the weighted sum of the negative loglikelihood summarization objective, and the binary

17

QA-retrieval limitations

Compositional Questions (-): Using label smearing percludes
answering questions that can be solved with > 1 QA-pairs.
▶ What are the primary causes of infertility in women that canbe addressed through preventive measures18

Work addressing compositional questions with QA-pairmemories▶ QA Is the New Knowledge Representation: Question-Answer
Pairs as Knowledge Bases[Chen et al.(2022a)Chen, Cohen, Jong, Gupta, Presta, Verga, and Wieting]▶ Augmenting Pre-trained Language Models with QA-Memory
for Open-Domain Question Answering[Chen et al.(2022b)Chen, Verga, Jong, Wieting, and Cohen]⊙ To answer q ′ , retrieve related QA pairs (q1 , a1 ) , . . . , (qk , ak )⊙ Combine retrieved QA pairs with original question q ′ .
⊙ Transformer to fuse this information and generate an answer▶ Effective usage of retrieved memories is an open problem

19

Memory Augmented Models: Summary▶ Memory augmented models have attributes that make them
particularly suitable for high-stakes domains▶ Attribution to sources, editing, memory scale, efficient QAOpen questions:▶ Choice of knowledge representation (entity embeddings?
passages? QA pairs? ...)▶ Effective usage of memories

20Role of ChatGPT in Healthcare QA▶ Generalization from fewer labeled examples; Conversational
interface; Multimodaliy;2

https://hai.stanford.edu/news/how-foundation-models-can-advance-ai-healthcare

21Case Study # 2: Programming

22

What is Program Synthesis?

▶ Program Synthesis: given a natural language specification,
generate the corresponding computer program⊙ Improve programmer productivity⊙ Make software development more accessible to non-experts⊙ Enable rapid prototyping

23

Example: Plotting
Visualizing data and functions▶ Task: Update plot from a natural language utterance

KB

Utterance

Plotter

Knowledge

plot action

24

(a)

212

Learning
Task
213

Figure 1: Illustration of two of the supported plot types, showing some
Plot type “line chart”.

214
215

Utterance: “use the log scale for the X-axis"
listings xcolor

216
217
218
219
220
221
222
223
224
225
226
227
228
229

1

import matplotlib.pyplot as plt
plt.plot(...)
plt.xscale(’log’)
plt.show()in Portable
3.3

Inpu

Dialog hist
can treat th4
that is we c
While this request is achieved by a single line
arate datapof code, line 3, some natural language requests
from three▶ Conversational:
plotslines
can ofbecodespecifiedmultiple-turns
require multiple
and ininvolvemultiby the text
ple slots. Figure ?? and Figure ?? illustrate thestate as rep
slots of a line chart, and a 3d surface plot, respecdialog histo
tively. Illustrations of all supported plot types andutterances.
their slots are in the supplementary material. Tabletion of the
1 lists all supported plot types and corresponding
25
slots.
2
3

Plot manipulation question: fonts

26

Plot manipulation question: vertical lines

27

Plot manipulation question (3/3): invert y-axis

28

Conversational Plotting Agent▶ A conversational plotting agent⊙ Knowledgable: has access to relevant knowledge⊙ Conversational: plots can be specified in multiple-turns▶ Framed it as a problem of goal-oriented dialog⊙ Slot-filling: fixed symbolic state representation

29

Slots: Line Chart

30

Slots: Pie Chart

31

Slots: 3D Surface

32

Shared Slots

33

1.

Plot Types
Axes

2.
3.

3D Surface
Bar Chart

4.

Contour/Filled

5.
6.

Contour/LinedHistogram

7.
8.

MatrixLine Chart

9.

Pie Chart

10.
11.

Polar
Scatter

12.

Streamline

Slots
Polarize, X-axis Scale, Y-axis Scale, X-axis Position, Y-axis Position, Invert X-axis, Invert Y-axis,
Grid Line Type, Grid Line Style,Grid Line Width, Grid Line Color, Font Size
Color map, Invert X-axis, Invert Y-Axis, Invert Z-Axis
Bar Orientation, Bar Height, Bar Face Color, Bar Edge Width, Bar Edge Color, Show Error Bar,
Error Bar Color, Error Bar Cap Size, Error Bar, Cap Thickness, Data Series Name
Contour Plot Type, Number of levels, Color Map, Color Bar Orientation, Color Bar
Length, Color Bar Thickness
Contour Plot Type, Lined Style, Line Width
Number of Bins, Bar Relative Width, Bar Face Color, Bar Edge Width, Bar Edge
Color, Data Series Name
Color Map, Invert X-axis, Invert Y-axis
Line Style, Line Width, Line Color, Marker Type, Marker Size, Marker Face Color,
Marker Edge Color, Marker Interval, Data Series Name, Show Error Bar, Error Bar Color,
Error Bar Cap Size, Error Bar Cap Thickness
Exploding Effect, Precision Digits, Percentage tags’ distance from center,Label tag’s distance from center, Radius, Section Edge Width, Section Edge Color
Polarize, Grid Line Type, Grid Line Style, Grid Line Width, Grid Line Color, Font Size
Polarize, Marker Type, Marker Size, Marker Face Color, Marker Edge Width, Marker Edge Color,
Color Map, Color Bar Orientation, Color Bar Length Color Bar Thickness
Density, Line Width, Line Color, Color Map, Arrow Size, Arrow Style

Table: Types and slots: all Matplotlib plot types are supported

34

Model35

Model Performance(Shao, & Nakashole, ACL 2020)▶

⊙ Human Performance: 80.6
⊙ our Model: 61.3

36

LLMs that Code37

ChatGPT Matplotlib(1/3): fonts

38

ChatGPT Matplotlib(2/3): vertical lines

39

ChatGPT Matplotlib(2/3): vertical lines: code works!

40

ChatGPT Matplotlib (3/3): invert y-axis

41

ChatGPT Matplotlib (3/3): invert y-axis, code works!

42

ChatGPT brings▶ Generalization ... not just Matplotlib, and not just Python▶ e.g., Tikz
humaninfants
eat

HiddenLayer 1

HiddenLayer 2

Output
Layer

x1
x2
milk
x3
x4

43

OpenAI Codex: behind GitHub Copilot

rXiv:2107.03374v2 [cs.LG] 14 Jul 2021

Evaluating Large Language Models Trained on CodeMark Chen * 1 Jerry Tworek * 1 Heewoo Jun * 1 Qiming Yuan * 1 Henrique Ponde de Oliveira Pinto * 1
Jared Kaplan * 2 Harri Edwards 1 Yuri Burda 1 Nicholas Joseph 2 Greg Brockman 1 Alex Ray 1 Raul Puri 1
Gretchen Krueger 1 Michael Petrov 1 Heidy Khlaaf 3 Girish Sastry 1 Pamela Mishkin 1 Brooke Chan 1
Scott Gray 1 Nick Ryder 1 Mikhail Pavlov 1 Alethea Power 1 Lukasz Kaiser 1 Mohammad Bavarian 1
Clemens Winter 1 Philippe Tillet 1 Felipe Petroski Such 1 Dave Cummings 1 Matthias Plappert 1
Fotios Chantzis 1 Elizabeth Barnes 1 Ariel Herbert-Voss 1 William Hebgen Guss 1 Alex Nichol 1 Alex Paino 1
Nikolas Tezak 1 Jie Tang 1 Igor Babuschkin 1 Suchir Balaji 1 Shantanu Jain 1 William Saunders 1
Christopher Hesse 1 Andrew N. Carr 1 Jan Leike 1 Josh Achiam 1 Vedant Misra 1 Evan Morikawa 1
Alec Radford 1 Matthew Knight 1 Miles Brundage 1 Mira Murati 1 Katie Mayer 1 Peter Welinder 1
Bob McGrew 1 Dario Amodei 2 Sam McCandlish 2 Ilya Sutskever 1 Wojciech Zaremba 1

Abstract

1. Introduction

Scalable sequence prediction models (Graves, 2014;
Vaswani et al., 2017; Child et al., 2019) have become a
We introduce Codex, a GPT language model finegeneral-purpose method for generation and representation
tuned on publicly available code from GitHub,
learning in many domains, including natural language proand study its Python code-writing capabilities.
cessing (Mikolov et al., 2013; Sutskever et al., 2014; Dai &A distinct production version of Codex powers
Le, 2015; Peters et al., 2018; Radford et al., 2018; Devlin
GitHub Copilot. On HumanEval, a new evaluaet al., 2018), computer vision (Van Oord et al., 2016; Menicktion set we release to measure functional correct& Kalchbrenner, 2018; Chen et al., 2020; Bao et al., 2021),
ness for synthesizing programs from docstrings,
audio and speech processing (Oord et al., 2016; 2018; Dhariour model solves 28.8% of the problems, while
wal et al., 2020; Baevski et al., 2020), biology (Alley et al.,
GPT-3 solves 0% and GPT-J solves 11.4%. Fur2019; Rives et al., 2021), and even across multiple modalithermore, we find that repeated sampling from theties (Das et al., 2017; Lu et al., 2019; Ramesh et al., 2021;
model is a surprisingly effective strategy for proZellers et al., 2021). More recently, language models haveducing working solutions to difficult prompts. Usalso fueled progress towards the longstanding challenge
ing this method, we solve 70.2% of our problems
of program synthesis (Simon, 1963; Manna & Waldinger,
with 100 samples per problem. Careful investiga1971), spurred by the presence of code in large datasetstion of our model reveals its limitations, including(Husain et al., 2019; Gao et al., 2020) and the resulting prodifficulty with docstrings describing long chainsgramming capabilities of language models trained on theseof operations and with binding operations to varidatasets (Wang & Komatsuzaki, 2021). Popular language
ables. Finally, we discuss the potential broader
modeling objectives like masked language modeling (Devlin
impacts of deploying powerful code generation3
et al., 2018) and span prediction (Raffel et al., 2020) havetechnologies,
covering
safety,security,
andeco[Chen et al.(2021)Chen, Tworek, Jun, Yuan, Pinto,
Kaplan,Edwards,
Burda, counterparts
Joseph, Brockman et al.]also been
adapted totrain their programming▶ Early analysis suggested GPT-3 could generate programs fromPython docstrings▶ Despite the fact that GPT-3 was not trained for code
generation
3

44

Codex▶ Synthesis challenge: given a Python docstring, generate thefunction implementation
Evaluating Large Language Models Trained on Code45

Evaluating language models for code
generation
Evaluating Large Language Models Trai

generate atlems. Thisbe selectedeach sample
tical in dep
highest mea
the problem

We conclud
broader imp
powerful co

2. Evalua

In this sectFigurefails
1. Pass
our models on the HumanEval dataset as a
▶ GPT-3
onrates
all ofproblems
framework.
function of model size. When a single sample is generated for each
▶ Fine-tuning on problems with this format of function synthesis
explain its aproblem, GPT-12B solves no problems, but Codex (fine-tuned(Codex-S)
improves
performance
Next, we don code) solves
28.8%
of the problems, and Codex-S (further
fine-tunedoncorrectly
implemented
standalonefunctions)solvescalled “Hum
▶ Sampling 100 programs, reranking and choosing best improves46

AlphaCode

RES EARCHCOMPUTER SCIENCE

the submitter
own tests or redebugging. B
to draw on s
Yujia Li*†, David Choi*†, Junyoung Chung†, Nate Kushman†, Julian Schrittwieser†, Rémi Leblond†,
previous cont
Tom Eccles†, James Keeling†, Felix Gimeno†, Agustin Dal Lago†, Thomas Hubert†, Peter Choy†,
are created fo
Cyprien de Masson d’Autume†, Igor Babuschkin, Xinyun Chen, Po-Sen Huang, Johannes Welbl,
programming
Sven Gowal, Alexey Cherepanov, James Molloy, Daniel J. Mankowitz, Esme Sutherland Robson,
International
Pushmeet Kohli, Nando de Freitas, Koray Kavukcuoglu, Oriol Vinyals*
petition (15) ain Informatics
Programming is a powerful and ubiquitous problem-solving tool. Systems that can assist programmers
are some of thor even generate programs themselves could make programming more productive and accessible.
in computer
Recent transformer-based neural network models show impressive code generation abilities yet still
thousands ofperform poorly on more complex tasks requiring problem-solving skills, such as competitive
world. Using p
programming problems. Here, we introduce AlphaCode, a system for code generation that achievedlenging froman average ranking in the top 54.3% in simulated evaluations on recent programming competitions on
vides a robust
the Codeforces platform. AlphaCode solves problems by generating millions of diverse programs using
many aspects
specially trained transformer-based networks and then filtering and clustering those programs to a
Early work u
maximum of just 10 submissions. This result marks the first time an artificial intelligence system haspetitive progr
performed competitively in programming competitions.
transformer mdigit solve ratea code genera
4utomaticallycreating
programs
given
aprogramming
problems
Python (13,14). ADal that
[Li et al.(2022)Li,
Choi,
Chung,
Kushman,
Schrittwieser,
Leblond,Eccles,inKeeling,
Gimeno,Lagomanages
et al.]

Competition-level code generation with AlphaCode▶ In 2022, DeepMind published AlphaCode, a system combining& expanding these ideas to solve competitive programming
problems
447

AlphaCode: Pipeline
RES EARCH | R E S E A R C H A R T I C L E

C48

AlphaCode: Ranking among contest participantsCompetition-Level Code Generation with AlphaCod(a) AlphaCode’s ranking in 10 contests(b)

Figure 1 | AlphaCode’s ranking on 10 simulated Codeforces contes
is better). AlphaCode ranked in the top 54.3% among contest partici
and achieved an estimated average rating of 1238. (a) shows the rat
49

al Are LLMs a job security threat to programmers?
ng codepen.

at to build

g rather than[Minelli et al, 2015]▶ Answer: not yet!
▶ Programming is more than writing lines of code:
problem-solving, strategic planning, andcreativity—abilities that, AI cannot fully replicate yet

50

LLMs that code: summary▶ (+) Works well on self-contained, short problems▶ (-) Public code repositories have lots of code with bugs
▶ (-) Generated code often has functional or security bugs. Stillneed to understand it!

51

Thanks
LLMs: Exciting opportunites for applying NLP to real world settings

52

Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor
Cai, Eliza Rutherford, Katie Millican, George van den
Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark,
Diego de Las Casas, Aurelia Guy, Jacob Menick, Roman Ring,
Tom Hennigan, Saffron Huang, Loren Maggiore, Chris Jones,
Albin Cassirer, Andy Brock, Michela Paganini, Geoffrey Irving,
Oriol Vinyals, Simon Osindero, Karen Simonyan, Jack W. Rae,
Erich Elsen, and Laurent Sifre. 2022.
Improving language models by retrieving from trillions oftokens.
In International Conference on Machine Learning, ICML 2022,
17-23 July 2022, Baltimore, Maryland, USA, pages 2206–2240.
Vannevar Bush et al. 1945.
As we may think.
The atlantic monthly, 176(1):101–108.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, HenriquePonde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri
Burda, Nicholas Joseph, Greg Brockman, et al. 2021.
52

Evaluating large language models trained on code.
arXiv preprint arXiv:2107.03374.
Wenhu Chen, William W. Cohen, Michiel De Jong, Nitish
Gupta, Alessandro Presta, Pat Verga, and John Wieting.
2022a.
Qa is the new kr: Question-answer pairs as knowledge bases.
arXiv preprint arXiv:2007.03819.
Wenhu Chen, Pat Verga, Michiel De Jong, John Wieting, andWilliam W. Cohen. 2022b.
Augmenting pre-trained language models with qa-memory for
open-domain question answerings.
arXiv preprint arXiv:2007.03819.
Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy.
2021.
Transformer feed-forward layers are key-value memories.
In Proceedings of the 2021 Conference on Empirical Methods
in Natural Language Processing, EMNLP 2021, Virtual Event /52

Punta Cana, Dominican Republic, 7-11 November, 2021, pages
5484–5495.
Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian
Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix
Gimeno, Agustin Dal Lago, et al. 2022.
Competition-level code generation with alphacode.
Science, 378(6624):1092–1097.

52