comment

Dual use of artificial-intelligence-powered drug
discovery
An international security conference explored how artificial intelligence (AI) technologies for drug discovery could
be misused for de novo design of biochemical weapons. A thought experiment evolved into a computational proof.

Fabio Urbina, Filippa Lentzos, Cédric Invernizzi and Sean Ekins

Risk of misuse

The thought had never previously struck
us. We were vaguely aware of security
concerns around work with pathogens or
toxic chemicals, but that did not relate tous; we primarily operate in a virtual setting.
Our work is rooted in building machinelearning models for therapeutic and toxic
targets to better assist in the design of newmolecules for drug discovery. We havespent decades using computers and AI toimprove human health—not to degrade
it. We were naive in thinking about thepotential misuse of our trade, as ouraim had always been to avoid molecularfeatures that could interfere with the manydifferent classes of proteins essential tohuman life. Even our projects on Ebola andneurotoxins, which could have sparked
thoughts about the potential negative
implications of our machine learningmodels, had not set our alarm bells ringing.
Our company—Collaborations
Pharmaceuticals, Inc.—had recentlyGenerated
compounds
LD50 dataset

10050

VX LD500O

400

VXCountt-SNE 2T

he Swiss Federal Institute for NBC(nuclear, biological and chemical)
Protection —Spiez Laboratory—
convenes the ‘convergence’ conference
series1 set up by the Swiss government toidentify developments in chemistry, biologyand enabling technologies that may have
implications for the Chemical and Biological
Weapons Conventions. Meeting every
two years, the conferences bring together
an international group of scientific anddisarmament experts to explore the current
state of the art in the chemical and biologicalfields and their trajectories, to think throughpotential security implications and to
consider how these implications can mosteffectively be managed internationally.
The meeting convenes for three days of
discussion on the possibilities of harm,should the intent be there, from cutting-edge
chemical and biological technologies.
Our drug discovery company received aninvitation to contribute a presentation on
how AI technologies for drug discovery
could potentially be misused.

N200SPO–50–100–500t-SNE 150000.1

0.2
0.3
0.4
Predicted LD50 (mg kg–1)

0.5

Fig. 1 | A t-SNE plot visualization of the LD50 dataset and top 2,000 MegaSyn AI-generated and predicted
toxic molecules illustrating VX. Many of the molecules generated are predicted to be more toxic in vivo inthe animal model than VX (histogram at right shows cut-off for VX LD50). The 2D chemical structure of VX
is shown on the right.

published computational machine learningmodels for toxicity prediction in differentareas, and, in developing our presentation tothe Spiez meeting, we opted to explore how
AI could be used to design toxic molecules.
It was a thought exercise we had notconsidered before that ultimately evolved
into a computational proof of concept formaking biochemical weapons.

Generation of new toxic molecules

We had previously designed a commercial
de novo molecule generator that we called
MegaSyn2, which is guided by machinelearning model predictions of bioactivity
for the purpose of finding new therapeutic
inhibitors of targets for human diseases.
This generative model normally penalizes
predicted toxicity and rewards predicted
target activity. We simply proposed to invert
this logic by using the same approach todesign molecules de novo, but now guiding
the model to reward both toxicity andbioactivity instead. We trained the AI withmolecules from a public database using acollection of primarily drug-like molecules(that are synthesizable and likely to beabsorbed) and their bioactivities. We opted
to score the designed molecules with anorganism-specific lethal dose (LD50) model3
and a specific model using data from thesame public database that would ordinarily

Nature Machine Intelligence | www.nature.com/natmachintell

be used to help derive compounds for thetreatment of neurological diseases (detailsof the approach are withheld but wereavailable during the review process). The
underlying generative software is built on,
and similar to, other open-source softwarethat is readily available4. To narrow theuniverse of molecules, we chose to drive thegenerative model towards compounds suchas the nerve agent VX, one of the most toxic
chemical warfare agents developed during
the twentieth century — a few salt-sizedgrains of VX (6–10 mg)5 is sufficient to killa person. Other nerve agents with the samemechanism such as the Novichoks have also
been in the headlines recently and used inpoisonings in the UK and elsewhere6.
In less than 6 hours after starting on ourin-house server, our model generated 40,000
molecules that scored within our desired
threshold. In the process, the AI designed
not only VX, but also many other known
chemical warfare agents that we identified
through visual confirmation with structuresin public chemistry databases. Many new
molecules were also designed that looked
equally plausible. These new molecules
were predicted to be more toxic, based on
the predicted LD50 values, than publicly
known chemical warfare agents (Fig. 1).
This was unexpected because the datasets
we used for training the AI did not include

comment
these nerve agents. The virtual moleculeseven occupied a region of molecularproperty space that was entirely separate
from the many thousands of molecules in
the organism-specific LD50 model, which
comprises mainly pesticides, environmental
toxins and drugs (Fig. 1). By inverting theuse of our machine learning models, we
had transformed our innocuous generativemodel from a helpful tool of medicine to agenerator of likely deadly molecules.
Our toxicity models were originallycreated for use in avoiding toxicity, enabling
us to better virtually screen molecules (forpharmaceutical and consumer productapplications) before ultimately confirming
their toxicity through in vitro testing. The
inverse, however, has always been true: thebetter we can predict toxicity, the better wecan steer our generative model to design
new molecules in a region of chemical
space populated by predominantly lethal
molecules. We did not assess the virtual
molecules for synthesizability or explore
how to make them with retrosynthesis
software. For both of these processes,
commercial and open-source software isreadily available that can be easily plugged
into the de novo design process of newmolecules7. We also did not physicallysynthesize any of the molecules; but witha global array of hundreds of commercial
companies offering chemical synthesis,that is not necessarily a very big step, andthis area is poorly regulated, with few if
any checks to prevent the synthesis of new,extremely toxic agents that could potentiallybe used as chemical weapons. Importantly,
we had a human in the loop with a firm
moral and ethical ‘don’t-go-there’ voice
to intervene. But what if the human wereremoved or replaced with a bad actor? Withcurrent breakthroughs and research into
autonomous synthesis8, a complete design–
make–test cycle applicable to making not
only drugs, but toxins, is within reach. Our
proof of concept thus highlights how anonhuman autonomous creator of a deadly
chemical weapon is entirely feasible.

A wake-up callWithout being overly alarmist, this shouldserve as a wake-up call for our colleagues
in the ‘AI in drug discovery’ community.
Although some domain expertise inchemistry or toxicology is still required
to generate toxic substances or biologicalagents that can cause significant harm, whenthese fields intersect with machine learningmodels, where all you need is the ability tocode and to understand the output of themodels themselves, they dramatically lowertechnical thresholds. Open-source machinelearning software is the primary route forlearning and creating new models like ours,
and toxicity datasets9 that provide a baselinemodel for predictions for a range of targets
related to human health are readily available.
Our proof of concept was focusedon VX-like compounds, but it is equallyapplicable to other toxic small moleculeswith similar or different mechanisms,with minimal adjustments to our protocol.
Retrosynthesis software tools are also
improving in parallel, allowing newsynthesis routes to be investigated for knownand unknown molecules. It is therefore
entirely possible that novel routes can bepredicted for chemical warfare agents,
circumventing national and international
lists of watched or controlled precursor
chemicals for known synthesis routes.
The reality is that this is not science
fiction. We are but one very small
company in a universe of many hundredsof companies using AI software for drug
discovery and de novo design. How manyof them have even considered repurposing,or misuse, possibilities? Most will workon small molecules, and many of thecompanies are very well funded and likely
using the global chemistry network to maketheir AI-designed molecules. How many
people have the know-how to find thepockets of chemical space that can be filledwith molecules predicted to be orders ofmagnitude more toxic than VX? We do notcurrently have answers to these questions.
There has not previously been significant
discussion in the scientific community about
this dual-use concern around the applicationof AI for de novo molecule design, at least
not publicly. Discussion of societal impactsof AI has principally focused on aspects
such as safety, privacy, discrimination andpotential criminal misuse10, but not on
national and international security. When
we think of drug discovery, we normally
do not consider technology misusepotential. We are not trained to consider
it, and it is not even required for machinelearning research, but we can now share
our experience with other companies
and individuals. AI generative machinelearning tools are equally applicable to larger
molecules (peptides, macrolactones, etc.)
and to other industries, such as consumer
products and agrochemicals, that also haveinterests in designing and making newmolecules with specific physicochemicaland biological properties. This greatly
increases the breadth of the potential
audience that should be paying attention
to these concerns.
For us, the genie is out of the medicine
bottle when it comes to repurposing ourmachine learning. We must now ask: what
are the implications? Our own commercialtools, as well as open-source software tools
and many datasets that populate public
databases, are available with no oversight.
If the threat of harm, or actual harm,
occurs with ties back to machine learning,
what impact will this have on how thistechnology is perceived? Will hype in thepress on AI-designed drugs suddenly flip toconcern about AI-designed toxins, public
shaming and decreased investment in thesetechnologies? As a field, we should open a
conversation on this topic. The reputationalrisk is substantial: it only takes one badapple, such as an adversarial state or otheractor looking for a technological edge, tocause actual harm by taking what we havevaguely described to the next logical step.
How do we prevent this? Can we lock away
all the tools and throw away the key? Dowe monitor software downloads or restrict
sales to certain groups? We could follow theexample set with machine learning models
like GPT-311, which was initially waitlist
restricted to prevent abuse and has an API
for public usage. Even today, without awaitlist, GPT-3 has safeguards in place toprevent abuse, Content Guidelines, a freecontent filter and monitoring of applications
that use GPT-3 for abuse. We know of norecent toxicity or target model publications
that discuss such concerns about dual use
similarly. As responsible scientists, we needto ensure that misuse of AI is prevented, andthat the tools and models we develop areused only for good.
By going as close as we dared, we
have still crossed a grey moral boundary,demonstrating that it is possible todesign virtual potential toxic molecules
without much in the way of effort, time orcomputational resources. We can easily erase
the thousands of molecules we created, but
we cannot delete the knowledge of how torecreate them.

Broader effects on society

There is a need for discussions acrosstraditional boundaries and multiple
disciplines to allow for a fresh look at AI
for de novo design and related technologies
from different perspectives and with a wide
variety of mindsets. Here, we give somerecommendations that we believe will
reduce potential dual-use concerns for AI
in drug discovery. Scientific conferences,such as the Society of Toxicology andAmerican Chemical Society, should actively
foster a dialogue among experts fromindustry, academia and policy making
on the implications of our computationaltools. There has been recent discussion inthis journal regarding requirements forbroader impact statements from authors
submitting to conferences, institutional

Nature Machine Intelligence | www.nature.com/natmachintell

comment
review boards and funding bodies as well as
addressing potential challenges12. Making
increased visibility a continuous effort
and a key priority would greatly assist in
raising awareness about potential dual-use
aspects of cutting-edge technologies andwould generate the outreach necessary tohave everyone active in our field engage in
responsible science. We can take inspirationfrom examples such as The Hague Ethical
Guidelines13, which promote a cultureof responsible conduct in the chemical
sciences and guard against the misuse ofchemistry, in order to have AI-focused
drug discovery, pharmaceutical andpossibly other companies agree to a codeof conduct to train employees, secure theirtechnology, and prevent access and potential
misuse. The use of a public-facing API formodels, with code and data available uponrequest, would greatly enhance security
and control over how published models are
utilized without adding much hindrance
to accessibility. Although MegaSyn is acommercial product and thus we havecontrol over who has access to it, goingforward, we will implement restrictions oran API for any forward-facing models. A
reporting structure or hotline to authorities,
for use if there is a lapse or if we becomeaware of anyone working on developing
toxic molecules for non-therapeutic uses,
may also be valuable. Finally, universities
should redouble their efforts toward theethical training of science students andbroaden the scope to other disciplines, andparticularly to computing students, so thatthey are aware of the potential for misuse of
AI from an early stage of their career, as wellas understanding the potential for broader
impact12. We hope that by raising awarenessof this technology, we will have gone someway toward demonstrating that although
AI can have important applications inhealthcare and other industries, we should
also remain diligent against the potential
for dual use, in the same way that we wouldwith physical resources such as molecules
or biologics.
❐Fabio Urbina1, Filippa Lentzos2,
Cédric Invernizzi 3 and Sean Ekins

1✉Collaborations Pharmaceuticals, Inc., Raleigh, NC,
USA. 2Department of War Studies and Departmentof Global Health & Social Medicine, King’s CollegeLondon, London, UK. 3Spiez Laboratory, Federal
Department of Defence, Civil Protection and Sports,Spiez, Switzerland.
✉e-mail: sean@collaborationspharma.com
1

Published: xx xx xxxx
https://doi.org/10.1038/s42256-022-00465-9
References

1. Spiez Convergence Conference https://www.spiezlab.admin.ch/en/home/meta/refconvergence.html (2021).
2. Urbina, F., Lowden, C. T., Culberson, J. C. & Ekins, S. https://doi.
org/10.33774/chemrxiv-2021-nlwvs (2021).
3. Mansouri, K. et al. Environ. Health Perspect. 129, 047013 (2021).
4. Blaschke, T. et al. J. Chem. Inf. Model. 60, 5918–5922 (2020).
5. National Research Council Committee on Toxicology. https://www.ncbi.nlm.nih.gov/books/NBK233724/ (National Academies
Press, 1997).

Nature Machine Intelligence | www.nature.com/natmachintell

6. Aroniadou-Anderjaska, V., Apland, J. P., Figueiredo, T. H., De
Araujo Furtado, M. & Braga, M. F. Neuropharmacology 181,108298 (2020).
7. Genheden, S. et al. J. Cheminform. 12, 70 (2020).
8. Coley, C. W. et al. Science 365, eaax1566 (2019).
9. Dix, D. J. et al. Toxicol. Sci. 95, 5–12 (2007).
10. Hutson, M. The New Yorker https://www.newyorker.com/tech/annals-of-technology/who-should-stop-unethical-ai (2021).
11. Brown, T. B. et al. Preprint at arXiv https://arxiv.org/abs/2005.14165(2020).
12. Prunkl, C. E. A. et al. Nat. Mach. Intell. 3, 104–110 (2021).
13. Organisation for the Prohibition of Chemical Weapons. The Hague
Ethical Guidelines https://www.opcw.org/hague-ethical-guidelines(2021).

Acknowledgements

We are grateful to the organizers and participants of theSpiez Convergence conference 2021 for their feedback andquestions. C.I. contributed to this article in his personal
capacity. The views expressed in this article are those of theauthors only and do not necessarily represent the position
or opinion of Spiez Laboratory or the Swiss Government.
We kindly acknowledge US National Institutes of Healthfunding under grant R44GM122196-02A1 from theNational Institute of General Medical Sciences and
1R43ES031038-01 and 1R43ES033855-01 from the National
Institute of Environmental Health Sciences for our machinelearning software development and applications. Research
reported in this publication was supported by the National
Institute of Environmental Health Sciences of the National
Institutes of Health under grants R43ES031038 and1R43ES033855-01. The content is solely the responsibilityof the authors and does not necessarily represent the official
views of the National Institutes of Health.

Competing interests

F.U. and S.E. work for Collaborations Pharmaceuticals,Inc. F.L. and C.I. have no conflicts of interest.

Additional information

Peer review information Nature Machine Intelligence
thanks Gisbert Schneider and Carina Prunkl for theircontribution to the peer review of this work.