Automated image-based taxon
identification using deep learningand citizen-science contributions
Miroslav Valan

Doctoral Thesis in Systematic Zoology at Stockholm University, Sweden 2021

Automated image-based taxon identification using
deep learning and citizen-science contributions
Miroslav Valan
Academic dissertation for the Degree of Doctor of Philosophy in Systematic Zoology at
Stockholm University to be publicly defended on Wednesday 10 March 2021 at 14.00 in Vivi
Täckholmsalen (Q-salen), NPQ-huset, Svante Arrhenius väg 20.
Abstract

The sixth mass extinction is well under way, with biodiversity disappearing at unprecedented rates in terms of species
richness and biomass. At the same time, given the currentpace, we would need the next two centuries to complete theinventory of life on Earthand this is only one of the necessary steps toward monitoring and conservation of species.
Clearly, there is an urgent need to accelerate the inventory and the taxonomic researchrequired to identify and describe
the remaining species, a critical bottleneck. Arguably, leveraging recent technological innovations is our best chance tospeed up taxonomic research. Given that taxonomy has been and still is notably visual, and the recent break-throughs incomputer vision and machine learning, it seems that the time is ripe to exploreto what extent we can accelerate morphologybased taxonomy using these advances inartificial intelligence. Unfortunately, these so-called deep learning systems often
requiresubstantial computational resources, large volumes of labeled training data and sophisticated technical support,
which are rarely available to taxonomists. This thesis is devoted to addressing these challenges. In paper I and paper II,
we focus on developing an easy-to-use (’off-the-shelf’) solution to automated image-based taxon identification, which isat the same time reliable, inexpensive, and generally applicable. This enables taxonomists to build their own automatedidentification systems without prohibitive investments in imaging and computation. Our proposed solution utilizes atechnique called feature transfer, in which a pretrained convolutional neural network (CNN) is used to obtain image
representations (”deep features”) for a taxonomic task of interest. Then, these features are used to train a simpler system,
such as a linear support vector machine classifier. In paper I we optimized parameters for feature transfer on a range ofchallenging taxonomic tasks, from the identification of insects to higher groups --- even when they are likely to belong
to subgroups that have not been seen previously --- to the identification of visually similar species that are difficult toseparate for human experts. In paper II, we applied the optimal approach from paper I to a new set of tasks, including a task
unsolvable by humans - separating specimens by sex from images of body parts that were not previously known to show
any sexual dimorphism. Papers I and II demonstrate that off-the-shelf solutions often provide impressive identification
performance while at the same time requiring minimal technical skills. In paper III, we show that phylogenetic information
describing evolutionary relationships among organisms can be used to improve the performance of AI systems for taxon
identification. Systems trained with phylogenetic information do as well as or better than standard systems in terms ofcommon identification performance metrics. At the same time, the errors they make are less wrong in a biological sense, andthus more acceptable to humans. Finally, in paper IV we describe our experience from running a large-scale citizen science
project organized in summer 2018, the Swedish Ladybird Project, to collect images for training automated identification
systems for ladybird beetles. The project engaged more than 15,000 school children, who contributed over 5,000 images
and over 15,000 hours of effort. The project demonstrates the potential of targeted citizen science efforts in collecting therequired image sets for training automated taxonomic identification systems for new groups of organisms, while providing
many positive educational and societal side effects.
Stockholm 2021
http://urn.kb.se/resolve?urn=urn:nbn:se:su:diva-189460
ISBN 978-91-7911-416-9
ISBN 978-91-7911-417-6

Department of Zoology
Stockholm University, 106 91 Stockholm

AUTOMATED IMAGE-BASED TAXON IDENTIFICATION USING
DEEP LEARNING AND CITIZEN-SCIENCE CONTRIBUTIONS

Miroslav Valan

Automated image-based taxon
identification using deeplearning and citizen-science
contributions
Miroslav Valan©Miroslav Valan, Stockholm University 2021
ISBN print 978-91-7911-416-9
ISBN PDF 978-91-7911-417-6
Printed in Sweden by Universitetsservice US-AB, Stockholm 2021

Dedicated to mydaughter Mia and son
Teodor without whom
this thesis would havebeen completed two
years ago.

Acknowledgment

I would like to express my deepest gratitude to my supervisor
Fredrik Ronquist for his patience, unconditional support andunderstanding. I am in his debt for supporting my choices andfor finding a way to finish this thesis without too much hassle.
I have received much support from my co-supervisors
Atsuto Maki, Karoly Makonyi and Nuria Albet-Tores. Also, I
would like to thank my numerous colleagues from the SU and
NRM and especially Savantic which I considered my second
home during the last five years.
I am very grateful to all my coauthors and those who in one
way or the other were involved in the work that resulted in thisthesis. Thank you all.
I am indebted to my wife Vlasta, for her love, patience,
encouragement, endless support, sacrifice and forgivenessduring the most important journey - life. I want to thank myparents, Dragica and Mirko and my brother Dragoslav.
Lastly and most importantly, I want to thank my children forbeing an inexhaustible source of joy and happiness. And thebest thing is that we are just starting.
To my 3 years old son Teodor for making me feel stronger,
more organized and better at prioritizing what matters themost in life. You taught me a lesson I once knew: You cansucceed only after you try, so never stop trying.
To my giggly and miraculous baby girl Mia. I love seeing you
happy for no obvious reason. So, just keep giggling.

Abstract
The sixth mass extinction is well under way, with biodiversity disappearing at unprecedented rates in terms of species richness and biomass. At the same time, given the current
pace, we would need the next two centuries to complete the inventory of life on Earthand this is only one of the necessary steps toward monitoring and conservation of species.
Clearly, there is an urgent need to accelerate the taxonomic research required to identify
and describe the remaining species. Arguably, leveraging recent technological innovations
is our best chance to speed up taxonomic research. Given that taxonomy has been and still
is notably visual, and the recent breakthroughs in computer vision and machine learning, it
seems that the time is ripe to explore to what extent we can accelerate morphology-basedtaxonomy using these advances in arti cial intelligence (AI). Unfortunately, these so-called
deep learning systems often require substantial computational resources, large volumes oflabeled training data and sophisticated technical support, none of which are readily available to taxonomists. This thesis is devoted to addressing these challenges. In paper I
and paper II, we focus on developing an easy-to-use ('o -the-shelf') solution to automatedimage-based taxon identi cation, which is at the same time reliable, inexpensive, and generally applicable. Such a system would enable taxonomists to build their own automatedidenti cation systems without advanced technical skills or prohibitive investments in imaging or computation. Our proposed solution utilizes a technique called feature transfer, inwhich a pretrained convolutional neural network is used to obtain image representations("deep features") for a taxonomic task of interest. Then, these features are used to train asimpler system, such as a linear support vector machine classi er. In paper I we optimized

1parameters for feature transfer on a range of challenging taxonomic tasks, from the identi cation of insects to higher groups {{{ even when they are likely to belong to subgroups
that have not been seen previously {{{ to the identi cation of visually similar species thatare dicult to separate for human experts. We nd that it is possible to nd a solution that
performs very well across all of these tasks. In paper II, we applied the optimal approachfrom paper I to a new set of tasks, including a task unsolvable by humans - separating
specimens by sex from images of body parts that were not previously known to show any
sexual dimorphism. Papers I and II demonstrate that an o -the-shelf solution can provide
impressive identi cation performance while at the same time requiring minimal technicalskills. In paper III, we show that information describing evolutionary relationships amongorganisms can be used to improve the performance of AI systems for taxon identi cation.
Systems trained with taxonomic or phylogenetic information do as well as or better thanstandard systems in terms of generally accepted identi cation performance metrics. At thesame time, the errors they make are less wrong in a biological sense, and thus more acceptable to humans. Finally, in paper IV we describe our experience from running a large-scale
citizen science project organized in summer 2018, the Swedish Ladybird Project, to collectimages for training automated identi cation systems for ladybird beetles. The project engaged more than 15,000 school children, who contributed over 5,000 images and over 15,000
hours of e ort. The project demonstrates the potential of targeted citizen science e orts incollecting the required image sets for training automated taxonomic identi cation systems
for new groups of organisms, while providing many positive educational and societal sidee ects.

2

Abstrakt
Vi a•r mitt inne i den sj•atte massutrotningen, och den biologiska m
angfalden f•orsvinner ien rasande fart. Arter f•orloras f•or alltid och och den totala biomassan minskar stadigt.
Samtidigt skulle vi beh•ova tv
ahundra 
ar f•or att slutf•ora inventeringen av livet p
a jordeni nuvarande takt, och detta a•r bara ett av de n•odv•andiga stegen mot •overvakning och
bevarande av den biologiska m
angfalden. Med tanke p
a detta star det klart att vi m
astef•ors•oka paskynda den taxonomiska forskning som kr•avs f•or att identi era och beskriva deaterstaende arterna. Att utnyttja de senaste 
arens tekniska framsteg a•r sannolikt v
ar b•astachans att g•ora detta. Med tanke p
a att taxonomi har varit och fortfarande a•r baserat till
tor del pa visuella karakt•arer, och att det har gjorts stora framsteg de senaste 
aren inom
datorseende och maskininl•arning, a•r det h•og tid att utforska i vilken utstr•ackning vi kan
accelerera morfologibaserad taxonomi med hj•alp av arti ciell intelligens (AI). De senaste
framstegen bygger p
a s
a kallad djupinl•arning (\deep learning"), vilket ofta kr•aver betydande ber•akningsresurser, stora volymer tr•aningsdata och avsev•ard teknologisk kompetens.
Dessa resurser a•r s•allan tillg•angliga f•or taxonomer. Forskningen som redovisas i denna
avhandling syftar till att avhj•alpa dessa problem. I uppsats I och uppsats II fokuserar vip
a att utveckla en l•attanv•and standardl•osning f•or automatiserad bildbaserad taxonidenti ering, som samtidigt •ar tillf•orlitlig, l•attillg•anglig och allm•ant till•amplig. Ett sadantstandardsystem skulle g•ora det m•ojligt f•or taxonomer att bygga sina egna automatiserade
identi eringssystem utan o•overkomliga investeringar i ber•akningsresurser eller i att ackumulera stora digitala bilddatabaser. Var l•osning anv•ander en teknik som bygger p
a att
extrahera de element eller egenskaper som uppfattas av ett avancerat neuralt n•atverk (ett3

\convolutional neural network") tr•anat f•or en generell bildklassi ceringsuppgift i bilder
avsedda f•or en annan uppgift, en taxonomisk identi eringsuppgift. De extraherade bildegenskaperna kan sedan anv•andas f•or att tr•ana ett enklare klassi ceringssystem, till exempel
en s
a kallad st•odvektormaskin (\support vector machine"). Vi optimerade parametrarnaf•or den h•ar typen av system p
a en rad utmanande taxonomiska uppgifter, fr
an identi ering av insekter till h•ogre taxa {{{ •aven n•ar de sannolikt tillh•or undergrupper som inte har
setts tidigare {{{ till identi ering av visuellt snarlika arter som a•r sv
ara att s•arskilja f•orm•anskliga experter. Vi fann att det var m•ojligt att utforma ett sadant system s
a att det
hade god prestanda f•or samtliga dessa uppgifter. I uppsat II anv•ande vi det optimala systemet fr
an papper I till en ny upps•attning uppgifter, inklusive en uppgift som inte kan l•osas
av m•anniskor - att separera hanar fr
an honor utifr
an bilder av kroppsdelar som inte tidigare var k•anda att visa nagon sexuell dimor sm. Uppsats I och emph II visar att det g
ar
att utveckla standardl•osningar som ger imponerande identi eringsprestanda hos de f•ardiga
identi eringssystemen och samtidigt kr•aver minimala tekniska f•ardigheter av anv•andaren.
I uppsats III visar vi att information som beskriver evolution•ara sl•aktskapsf•orh
allandenmellan organismer kan anv•andas f•or att f•orb•attra prestandan hos AI-system f•or taxonomisk
identi ering. System tr•anade med taxonomisk eller fylogenetisk information presterar lika
bra som eller b•attre •an standardsystem n•ar de utv•arderas med allm•ant accepterade prestandam
att. Samtidigt a•r felen de g•or mindre felaktiga i biologisk mening och d•armed
mer acceptabla f•or m•anniskor. Slutligen beskriver vi i uppsats IV v
ar erfarenhet av att
genomf•ora ett storskaligt medborgarvetenskapligt projekt som anordnades sommaren 2018,
Nyckelpigef•ors•oket, f•or att samla in bilder f•or att tr•ana AI-system f•or identi ering av nyckelpigor. Projektet engagerade mer a•n 15 000 skolbarn, som bidrog med o•ver 5,000 bilderoch •over 15,000 timmars arbete. Projektet visar vilken enorm potential som nns i att
engagera medborgarforskare i att samla in de n•odv•andiga bilderna f•or att kunna tr•ana
AI-system f•or automatisk identi ering av nya grupper av djur och v•axter. Samtidigt kansadana projekt ge manga positiva bie ekter. Inte minst kan de v•acka allm•anhetens nykenhet inf•or den biologiska m
angfalden och intresset f•or att bevara den f•or framtiden.

4Author's contributions
The thesis is based on the following articles, which are referred to in the text
by their Roman numerals:

I. Valan, M., Makonyi, K., Maki, A., Vondracek, D., & Ronquist, F. (2019). Automated taxonomic identi cation of insects with expert-level accuracy using e ective featuretransfer from convolutional networks. Systematic Biology, Volume 68, Issue 6, November
2019, Pages 876{895, https://doi.org/10.1093/sysbio/syz014.
II. Valan, M., Vondracek, D., & Ronquist, F. Awakening taxonomist's third eye:
exploring the utility of computer vision and deep learning in insect systematics. Submitted.
III. Valan, M., Nylander A. A. J. & Ronquist F. AI-Phy: improving automated imagebased identi cation of biological organisms using phylogenetic information. Manuscript.
IV. Valan, M., Bergman M., Forshage M. & Ronquist F. The Swedish Ladybird
Project: Engaging 15,000 school children in improving AI identi cation of ladybird beetles.
Manuscript.

5

Candidate contributions to thesis articles*Type of contribution

paper I

paper II

paper III

paper IV

Conceived the study

A

A

A

A

Designed the study

A

A

A

A

Collected the data

A

A

A

A

Analyzed the data

A

A

A

A

Manuscript preparation

A

A

A

A

Table 1: Contribution Explanation:
A - Substantial: took the lead role and performed the majority of the work.
B - Signi cant: provided a signi cant contribution to the work
C - Minor: contribution in some way, but contribution was limited.

6

Contents
Abstract1

Author contributions to thesis articles

5

Abbreviations

9

1 Introduction

10

1.1

Convolutional neural networks and deep learning. . . . . . . . . . . . . .

13

1.2

Aim of the current thesis . . . . . . . . . . . . . . . . . . . . . . . . . . . .

16

2 Summary of papers
2.1

2.2

2.3

2.4

21

Paper I . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

21

2.1.1

Material and methods . . . . . . . . . . . . . . . . . . . . . . . . .

21

2.1.2

Datasets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

22

2.1.3

Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

22

2.1.4

Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

26

Paper II . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

28

2.2.1

Material and methods . . . . . . . . . . . . . . . . . . . . . . . . .

28

2.2.2

Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

31

Paper III . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

33

2.3.1

Material and methods . . . . . . . . . . . . . . . . . . . . . . . . .

33

2.3.2

Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

35

Paper IV . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

36

7

2.4.1

Design of the study. Material and methods . . . . . . . . . . . . . .

36

2.4.2

Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

37

3 Discussion

40

4 Concluding remarks and thoughts on not so far future

48

References

538

Abbreviations
A list of abbreviations used in the thesis:
AI - Arti cial Intelligence
ATI - Automated Taxonomic Identi cation aystem
CAM - Class Activation Maps
CNN - Convolutional Neural Network
CS - Citizen Science
DL - Deep Learning
GBIF - Global Biodiversity Information Facility
GPU - Graphical Processing Unit
FC - Fully Connected
LR - Logistic Regression
LS - Label Smoothing
PLS - Phylogenetic Label Smoothing
SLP2018 - Swedish Ladybird Project
SVC - Support Vector Classi er
SVM - Support Vector Machin
TLS - Taxonomic Label Smoothing

9

Chapter 1
Introduction
An understanding of the natural world
and what's in it is a source of not only
a great curiosity but great ful llment.
David AttenboroughBiodiversity is under unprecedented pressure due to climate change and the in uenceof humans. Based on the alarming rates at which species are disappearing it is more thanobvious that the sixth mass extinction is under way (Ehrlich, 1995; Laliberte and Ripple,
2004; Dirzo et al., 2014; Ripple et al., 2014; Maxwell et al., 2016; Ceballos et al., 2017).
Precious life forms are lost before we became aware of their existence; forms that took
evolution millions of years to create. If we would know what we have and what we may
lose it would be easier to convince decision-makers to take appropriate action to stop thisdevastating loss of biodiversity.
The scienti c eld charged with the task of describing and classifying life on Earth istaxonomy, an endeavor that is as old as humans. Since the very beginnings, we aimed tounderstand the World around us; we observed, compared, tried to understand and madesome conclusions; then we passed the knowledge on to the coming generations in oral
and later in written form. It is easy to imagine how food (i.e. living beings - plants,
animals and fungi) was on top of our priorities; we had to learn what is edible and tasty
10and what not, so we probably relied on some information about anatomical features todistinguish one form of life from another. Years later, this became more structured so
di erent forms of life started to be compared based on the same body parts, or the absenceor presence of some morphological structures. The rst written descriptions of di erent
species were composed by compiling such observations of characters. This is consideredas the beginning of descriptive taxonomy. During the 18th century, Carl Linnaeus, aSwedish botanist, zoologist and taxonomist, established universally accepted conventions
for classifying nature within a nested hierarchy and for the naming of organisms. Today,
this system is still in use and it is known as Linnaean taxonomy or modern taxonomy.
Taxonomy remained predominantly descriptive until the mid-20th century when it became more quantitative thanks to the developments in statistics. Data such as length,
width, angles, counts and ratios, combined with multivariate statistical methods, provided
a deeper understanding of patterns in the biological world. This marked the beginning oftraditional morphometrics (Marcus, 1990). In 1980's, taxonomist applied approaches toquantify and analyse variations in shape (known as geometric morphometrics Rohlf andMarcus (1993)), which was based on coordinates of outlines or landmarks. These were
useful for graphical visualisation and/or statistical analyses, but they were also used inbuilding some of the rst systems for automated taxon identi cation (see below).
Throughout its historical development, it has become increasingly clear that taxonomy
is more than just a descriptive scienti c discipline; it is a fundamental science on which
other sciences|such as ecology, evolution and conservation|rely. In an important sense,
taxonomy represents the World's scienti c frontier, marking the boundary between the
known and the unknown in our discovery of life forms. Unfortunately, taxonomic researchis still slow in expanding this frontier. At the current pace, it is expected that it will takemany years to describe all species of biological organisms on the planet. The gaps in our
taxonomic knowledge and the shortage of taxonomic expertise is known as the taxonomic
impediment (Agnarsson and Kuntner, 2007; Walter and Winterton, 2007; Rodman and
Cody, 2003; Ebach et al., 2011; Coleman, 2015). Clearly, accelerating taxonomic researchwould bring many positive e ects on a wide range of immensely important decisions our
11civilization needs to make in the very near future.
One possible approach to combating the taxonomic impediment would be to build sophisticated automated taxon identi cation systems (ATIs). ATIs could help in two ways.
First, they could take care of routine identi cations, freeing up the time of taxonomic
experts so that they could focus on more challenging and critical tasks in expanding ourknowledge of biodiversity. Second, sophisticated ATIs could also directly help in the process of identifying and describing new life forms. Until recently, however, ATIs were notparticularly e ective in solving these tasks. An important reason for this is that they
were based on hand-crafted features. For example, if the purpose were to identify insects,
relevant features might be the wing venation patterns, the positions of wing vein junctions, or the outlines of the whole body. After human experts identi ed some potentiallyinformative features, these features would then have to be identi ed in images manuallyor through automated procedures that were speci cally designed for the task at hand (Arbuckle et al., 2001; Feng et al., 2016; Francoy et al., 2008; Gauld et al., 2000; Lytle et al.,2010,?; O'Neill, 2007; Schr•oder et al., 1995; Steinhage et al., 2007; To lski, 2007, 2004;
Watson et al., 2003; Weeks et al., 1999a,b, 1997). Some of the ATIs developed using thesetechniques have shown great performance (Martineau et al., 2016), but the approach isdicult to generalize because it requires knowledge of programming and image analysis(to formalize manual or code automatic procedures for feature extraction), of machinelearning (to build an appropriate classi er) and of the task itself (expertise on the taxa ofinterest). Clearly, this approach does not generalize well. For every new task we need toconsider factors that determine the best target features, and then hand-craft procedures
to encode those features. For these reasons, such ATIs have been presented for only a few
groups. Note that a considerable amount of human e ort must be spent before we can even
evaluate whether it is feasible to solve the identi cation task at hand using this approach.

12

1.1

Convolutional neural networks and deep learning

In recent years, more general approaches to image classi cation have developed greatly(LeCun et al., 2015; Schmidhuber, 2015). This is part of a general trend in computerscience towards more sophisticated and intelligent systems, that is, towards more sophisticated arti cial intelligence (AI). The trend is driven by improved algorithms, rapidly
increasing amounts of data, and faster and cheaper computation. In the eld of computer
vision, the development has been particularly fast in recent years with the introduction of
more complex and sophisticated arti cial neural networks, known as convolutional neuralnetworks (CNNs), and the training of advanced (deep) versions of these networks withmassive amounts of data, also known as deep learning (DL). The dramatic progress incomputer vision has been enabled also by the development of graphical processing units(GPUs), adding a considerable amount of cheap processing power to modern computer
systems.
The rst super-human performance of GPU-powered CNNs in an image classi cation
task (Ciresan et al., 2011) was reported in 2011 in a trac sign competition (Stallkamp
et al., 2011). The breakthrough came in 2012, when a CNN architecture called AlexNet(Krizhevsky et al., 2012) out-competed all other systems in the ImageNet Large Scale
Visual Recognition Challenge (Russakovsky et al., 2015), a larger and more popular image
classi cation challenge. The good news about DL performance spread quickly, and we soon
witnessed successful applications in other research areas, such as face veri cation (Taigman
et al., 2014), object localisation (Tompson et al., 2015), image and video translation into
natural language (Karpathy and Fei-Fei, 2015), language translation (Sutskever et al., 2014;
Jean et al., 2015), speech recognition (Sainath et al., 2013; Hinton et al., 2012; Zhang andZong, 2015) and question-answer problems (Kumar et al., 2016).
The core of every CNN architecture is a set of convolutional (conv) layers, hence thename convolutional neural network (Fukushima, 1979, 1980; Fukushima et al., 1983; LeCunet al., 1989). The convolutional part of a CNN enables automatic feature learning; itworks as a \feature extractor". The resulting features are then fed through one or more
13

fully connected (FC) layers, which deal with the classi cation task. The FC layers inprinciple correspond to a traditional multi-layer perceptron (Rosenblatt, 1957) which is a
simple fully-connected feed-forward arti cial neural network. Learning in a CNN is possible
thanks to the backpropagation algorithm (Kelley, 1960; Linnainmaa, 1976; Werbos, 1982;
Rumelhart et al., 1986; Schmidhuber, 2014) and gradient-based optimization (Robbins
and Monro, 1951; Kiefer et al., 1952; Bottou et al., 2018). Most of the CNNs used today
also contain other layers, such as pooling (dimensionality reduction) (Fukushima, 1979,1980), normalization (e.g. BatchNorm (Io e and Szegedy, 2015), helps with stabilizing
the training) or regularization layers (Hanson, 1990; Srivastava et al., 2014) (helps withaddressing the over- tting), among many others.

Figure 1.1: Architecture of VGG16 (Simonyan and Zisserman 2014), a simple modern
CNN. VGG16 consists of ve convolutional blocks, each block consisting of two or three
convolutional layers (green) followed by a MaxPooling layer (red). These blocks are followed by three layers of fully connected neurons (gray), the last of which consists of avector of length 1000. Each element in this vector corresponds to a unique category in
the ImageNet Dataset (Russakovsky et al., 2015) for which this architecture was initiallybuilt. Adopted from paper I
To better understand the basic structure of a CNN, consider Figure 1.1 illustrating
a well known deep CNN architecture, VGG16 (Simonyan and Zisserman, 2014). This14

architecture is simple, yet very powerful and therefore one of the best studied. It has also
become one of the most commonly utilized architectures for addressing various researchquestions. VGG16 consists of ve convolutional blocks followed by two FC hidden layers
and the output layer (also FC), where the number of nodes corresponds to the numberof categories the network is trained for. The convolutional block is made of convolutional(conv) layers followed by a MaxPooling layer. Every conv layer in the VGG family is madeof 3x3 lters. The number of layers in each block and the number of lters in each layer vary,
so we have 2x64, 2x128, 3x256, 3x512, and 3x512 "layers x lters" respectively for the ve
convolutional blocks. Note that some of the recent CNNs have more than a hundred layers
including dozens of convolutional layers with much more complex architectures. VGG16
uses max pooling with kernel of size 2x2 and a stride of 2, taking only the maximum value
within the kernel (other options would be the average, sum, etc). This results in reduced
width and height of the feature matrix by a factor of two, and total amount of data bya factor of four. Unlike a conv layer, where nodes are connected to the input image or
previous layer only by the local region of the same size as the corresponding kernel, thenodes in a fully connected layer (FC) are connected to every node in the previous layer (asin a simple multi-layer perceptron).
Modern CNNs often require large sets of labeled images for successful supervised learning. Recently, it has been discovered that features learned by a CNN that has been trainedon a generic image classi cation task (source task) can be bene cial in solving a more specialized problem (target task) using a technique called transfer learning (Caruana, 1995;
Bengio, 2012; Yosinski et al., 2014; Azizpour et al., 2016)). Transfer learning works primarily because a fair amount of relevant low-level features (edges, corners, etc.) are likely
similar between source and target tasks. Intermediate (you can think of eye, nose, mouth,etc.) and high-level (e.g. head, leg) features are more specialized and their usefulness
depends on the distance between the source and target tasks.
There are two variants of transfer learning: feature transfer and ne-tuning. In featuretransfer, a pretrained CNN serves as an automated feature extractor (Azizpour et al.,2016; Donahue et al., 2014; Oquab et al., 2014; Sharif Razavian et al., 2014; Zeiler and15

Fergus, 2014; Zheng et al., 2016)). Each image is fed through a pretrained CNN, and its
representation (feature vector) is extracted from one of the layers of the CNN, capturinglow- to high-level image features. Then, these features are used to train a simpler machinelearning system, such as a Support Vector Machine (SVM) (Cortes and Vapnik, 1995)),
a Logistic Regression (LR) (Cox, 1958), a Random Forest (Breiman, 2001) or a Gradient
Boosting (Friedman, 2001). This approach is usually computationally more ecient andit can bene t from properties of the chosen classi er (e.g. SVMs tend to be resistant toover tting, outliers and class imbalance, while LR is simple, intuitive and ecient to train).
Taking a pretrained CNN (or part of it) as initialization for training a new model isknown as ne-tuning. Fine-tuning tends to work well when the specialized task is similar
to the original task (Yosinski et al., 2014). Compared to training a CNN from scratch,ne-tuning reduces the hunger for data and improves convergence speed, but it may requirea fair amount of computational power. In ne-tuning, the images have to be run throughthe CNN in a forward pass, and then the computed derivatives from the predictions have to
be backpropagated to modify the lters (the latter is the more computationally expensivepart). This process of alternating forward and backward passes has to be repeated until
our model converges. There is also the problem of de ning appropriate learning hyperparameters in order to enable sucient exibility in learning of the new task while avoiding
over tting.

1.2

Aim of the current thesis

With the breakthroughs in deep learning and computer vision outlined above, it is now
possible to meet the requirements for highly competent ATIs (Wu et al., 2019; Hansen
et al., 2020; Joly et al., 2018; Van Horn et al., 2018; Cui et al., 2018), which can help
accelerate taxonomic research. Given a sucient number of training examples and theirlabels (e.g. a species name obtained from a taxonomic expert or with DNA sequencing),
these new systems learn to identify features important for identi cation directly fromimages, without any interference from humans; that is, there is no need for an expert
16

to indicate what is informative, the system

nds the relevant image features by itself.

However, as indicated above, a limiting factor is access to sucient amounts of training
data, which could be a serious challenge for most species identi cation tasks. There arevarious reasons for this. Firstly, the species abundances are usually imbalanced: there
are often a few common species, while the majority of species are rarely seen and almost
never photographed (or collected). Secondly, the number of images, for those species thatare photographed (or collected), is hugely imbalanced towards more attractive groups or
subgroups. Among insects, for instance, butter ies are wildly popular targets for nature
photographers, while small midges, ies or parasitic wasps are almost never photographedregardless of how common they are. The popularity may also vary among morphs orlife stages; for instance, butter y eggs and pupae are photographed much less than adultbutter ies. Thus, collecting enough images of all species and relevant morphs to be able
to train a state-of-the-art AI system may be a daunting task. In addition to the challengeof putting together an adequate training set, another serious challenge in training sucha state-of-the-art AI system on a dedicated taxonomic task is that it requires advanced
technical skills that most taxonomists lack. In this thesis, I address these challenges.
The main focus of the thesis has been on insects because they are diverse, challenging
to identify and there are many groups of insects that are poorly studied. In fact, morethan half of the known species on Earth are insects (over a million according to Zhang(2011)); and many scientists are suggesting that what we know today is just a fraction ofwhat is left to be discovered (Mora et al., 2011; Stork et al., 2015; Novotny et al., 2002).
For illustration, consider estimates of the number of undescribed species of all chordates
together (15,000), plants (80,000) and insects (4,000,000) (Chapman et al., 2009). The
disparity is even greater if we base the comparison on how much we know about their
physiology, behaviour, spatial and temporal distributions. Despite these knowledge gaps,
insects play many important roles in our ecosystems, both bene cial ones, for instanceas pollinators of crops, but also less favorable ones, for instance as pests, invasive species
or even vectors of disease. The enormous diversity of insects, the shortage of taxonomic
expertise (Gaston and May, 1992; Gaston and O'Neill, 2004), and the importance of many17

insect species in our ecosystems combine to emphasize the need for accelerating taxonomic
research on insects and the potential use for ATIs in doing so. Nevertheless, the ndingspresented in the thesis are general and should apply to image-based identi cation of anygroup of organisms with AI.
An important goal of the current thesis has been to develop techniques enabling taxonomists to build their own sophisticated ATIs using reliable and computationally inexpensive approaches, and without prohibitive investments in imaging (paper I and II). In
paper I, we explored methods that might allow taxonomists to develop ATI systems evenwhen the available image data and machine learning expertise are limited. Speci cally,
we focused on feature transfer, as previous work has indicated that features obtained frompretrained CNNs is a good starting point for most visual recognition tasks (Sharif Razavian et al., 2014). A CNN pretrained on a general image classi cation task was used as
an automated feature extractor, and the extracted features were then used in training asimpler classi cation system for the taxonomic task at hand. By optimizing the feature
extraction protocol, we were able to develop high-performing ATIs for a range of taxonomic identi cation tasks using fairly limited image sets as training data. Speci cally, we
looked at two challenging types of tasks: (1) identi cation of insects to higher groups, evenwhen they are likely to belong to subgroups that have not been seen previously; and (2)identi cation of visually similar species that are dicult to separate for human experts.
For the rst type of task, we looked at the identi cation of images from the iDigBio repository of Diptera and Coleoptera, respectively, to higher taxonomic groups. For the second
type of task, we looked at the identi cation of beetles of the genus Oxytherea to species,
using a dataset assembled for the paper, and stone y larvae to species, using a previously
published dataset.
In paper II, we aimed to address some questions on automated identi cation that are
frequently asked by insect taxonomists: Which techniques are best suited for a quick starton an ATI project? How much data is needed? What is the needed image resolution? Is
it possible to tackle identi cation tasks that are unsolvable by humans? To answer thesequestions, we created two novel datasets of 10 visually similar species of the ower chafer18

beetle genus Oxythyrea. The best performing system found in paper I was then used as
an o -the-shelf solution and applied to these datasets in several experiments designed toanswer the questions. In addition, we repeated the same experiments using some stateof-the-art approaches in image recognition. We show that our o -the-shelf system, whileo ering an "easy-to-use instant-return" approach, is often sucient for testing interesting
hypotheses. In fact, the identi cation performance of ATIs based on the o -the-shelf
system was not too far from that of state-of-the-art approaches in our experiments, and it
provided similar insights (feasibility, misidenti cation patterns, etc.) compared to the moreadvanced systems. We even demonstrate that our o -the-shelf approach can be successfully
used on a challenging task that appears unsolvable to humans.
It is well known that CNNs occasionally make catastrophic errors; e.g., misidentifying
one category for a completely unrelated category - a mistake that humans would be very
unlikely to make. We address this in a biological setting in paper III by leveraging a
recently introduced technique called label smoothing (Szegedy et al., 2016). Speci cally,
we propose label smoothing based on taxonomic information (taxonomic label smoothing)or distances between species in a reference phylogeny (phylogenetic label smoothing). We
show that networks trained with taxonomic or phylogenetic information perform at least
as well on common performance metrics as standard systems (accuracy, top3 accuracy,
f1 score macro), while making errors that are more acceptable to humans and less wrong
in an objective biological sense. We validated our proposed techniques on two empiricalexamples (38,000 outdoor images of 83 species of snakes, and 2,600 habitus images of 153
species of butter ies and moths).
As mentioned above, CNNs typically require large training sets of accurately labeledimages. Assembling such training sets for developing ATIs could be addressed by soliciting
the help from citizen scientists. We explored this in paper IV. In the Swedish Ladybird
Project (SLP2018), we engaged more than 15,000 Swedish school children in collecting
photos of ladybird beetles (Coccinellidae). The children collected more than 5,000 photosof 30 species of coccinellids. This is almost as many coccinellid images as the rest of theWorld contributed from around the globe to the Global Biodiversity Information Facility
19

(GBIF) portal during the same period{{{the summer of 2018. We found that adding the
SLP2018 images to the GBIF data resulted in improvements of ATI model performanceacross various evaluation metrics for all but the most common ladybird species.

20

Chapter 2
Summary of papers
Begin at the beginning," the Kingsaid, very gravely, "and go on till youcome to the end: then stop.
Lewis Carroll

2.1

Paper I

2.1.1

Material and methods

Our experiments in paper I were designed to nd optimal feature extraction settings forvarious taxonomic identi cation tasks and training datasets using a single feed-forward passthrough a pretrained CNN. Recent work has indicated that these so-called deep features,
although the extraction of them has been learned on a general image classi cation task, arevery robust and, in combination with simple classi ers such as SVMs (Cortes and Vapnik,
1995), can yield results on par with or better than state-of-the-art results obtained withhand-crafted features (Azizpour et al., 2016; Sharif Razavian et al., 2014; Donahue et al.,2014; Oquab et al., 2014; Zeiler and Fergus, 2014; Zheng et al., 2016).
A well known CNN architecture, VGG16 (Simonyan and Zisserman, 2014), and its
publicly available checkpoint pretrained on the ImageNet task (Simonyan and Zisserman,21

2014), were utilized across all our experiments. Our experiments were based on features
extracted after each conv block, and we refer to them as c1-c5, respectively. The FC layers
were excluded because they were dependent on the image input size. In our experiments,
we investigated the e ects of: input image size, pooling strategy (Max vs Average), featuresfrom di erent layers (feature depth), normalization (l2 and/or signed square root), featurefusion, non-global pooling and image aspect ratio.

2.1.2

Datasets

To nd optimal hyperparameters for feature extraction we created four datasets representing two types of challenging taxonomic tasks; (1) identifying insects to higher groups
when they are likely to belong to subgroups that have not been seen previously; and (2)identifying visually similar species that are dicult to separate even for experts.
Three out of four datasets (D1-D4) were assembled speci cally for this paper (Table
2.1). The rst two datasets (head view of ies and top view of beetles, D1 and D2 respectively) were designed to investigate how far this approach can get us when assigning
novel species to known higher taxonomic categories. The remaining two datasets were used
to investigate whether the same techniques would be able to discriminate among visually
very similar species (top view of sibling beetle species and species of Plecoptera larvae indi erent life stages, D3 and D4 respectively). Images from all four datasets were taken
in lab settings. They all had uniform background (the same uniform background acrossall images in D3-D4) and with small amounts of image noise (pins, dust, labels, scales,measurements). In all datasets but D4, objects were large, centered and share almost thesame object orientation (imaged in a standard taxonomic imaging procedure).

2.1.3

Experiments

Impact of image size. Previous work demonstrated that concatenating features fromimages of di erent scales (image sizes) could improve the performance on

ne-grainedclassi cation tasks (Takeki et al., 2016). However, in order to obtain features from the22

Table 2.1: Datasets used in paper I. Datasets D1 and D2 are used for a task of assigning
novel species to known higher taxonomic categories and the other two datasets for a taskof separating specimens of visually similar species. In all datasets, the images were taken
in lab settings with uniform background, large centered objects (not in all images in D4);
same object orientation (except D4) and small amount of background noise (pins, dust,
labels, scales, measurements). Stars (*) indicate datasets composed for paper I using
images obtained from www.idigbio.org. Adapted from paper I.

ID

InsectCategories

Images per taxa

View

SourceD1

Flies

11 families

24 -159

face

*

D2

Beetles

14 families

18 - 900

top

*

D3

Beetles

3 species

40-205

topThis study

D4 Stone ies

9 species

107-505

top

Lytle et al. (2010)

same image of di erent scales one needs to execute multiple feed-forward passes which
results in increased computational cost. Unlike this technique, we opted for nding the
optimal input size for a single feed-forward pass. In this experiment we restricted ourattention to c5.
Impact of pooling strategy. Global pooling (Lin et al., 2013) is a common way to
reduce dimensionality of deep features. Despite several recently proposed alternatives,
the two most common pooling strategies are still global max pooling and global average
pooling. We experimented with both pooling strategies and with a simple combination of
the two (concatenation). As in the previous experiment, we used c5 features only.

23Impact of feature depth. According to Azizpour et al. (2016), one of the most important factors for the transferability of pretrained features is the distance between the target
and the source tasks. If the task is to separate breeds of dogs then we may expect thelayers toward the end (FC layers) to perform the best. This is because the source dataset
ImageNet has a lot of dog categories so the later layers have probably learned so-calledhigh level features (you can think of body parts and their shapes - legs, head). In contrast,
if the task is to separate two visually similar beetle species that di er only in small details,
such as the degree of hairiness (corresponding to ne-grained di erences in image texture),
then we may want to focus on features from earlier layers (conv layers). To investigate how
the feature depth a ects performance on our taxonomic identi cation tasks, we comparedextracted features from all ve convolutional blocks c1-c5.
Impact of feature normalization. Reducing the variance of the elements in the feature
vectors is known to facilitate classi cation. We experimented with two common normalization techniques: l2 -normalization and signed squared root normalization as in Arandjelovic
and Zisserman (2013).
Impact of feature fusion. The advantage of combining features from di erent layers isdemonstrated in Zheng et al. (2016). Unlike their work, we only tested fusion of featuresfrom conv blocks (c1-c5 ) to avoid dependency on image input size.
Impact of non-global pooling. Feature matrices of the intermediate layers are large.
The total size is equal to HxWxF - where H is the height, W is the width and F is the depthor the number of lters of the convolutional block. As the rst two dimensions (H,W) of
the feature matrices depend on image input size, and the number of lters is large, somedimensionality reduction is necessary in extracting features from intermediate conv layers.
Global pooling decreases the feature matrix to a vector of size 1x1xF. This minimizes thecomputational cost for classi er training, prevents over tting, but it is also known to result
in better performance compared to just attening raw feature matrices (Zheng et al., 2016).

24

We investigated the e ect of intermediate levels of dimensionality reduction. Speci cally,
we reduced raw feature matrices to matrices of sizes 2x2xF, 4x4xF, 7x7xF, 14x14xF and
28x28xF, which were then attened. These intermediate levels of dimensionality reduction
increase computational cost but potentially preserve more information.
Impact of image aspect ratio. We maintained the image aspect ratio across all theexperiments described above. The images were symmetrically padded with random or
uniform pixels, which resulted in preserved object aspect ratio but some loss of information
due to the added uninformative pixels. An alternative procedure would be to instead
preserve the image information by image resizing, resulting in distorted objects, instead of
padding with uninformative pixels. In this experiment, we compared these two approaches
to examine whether it was more important to maintain aspect ratio or to preserve image
information.
Classi er and evaluation of classi cation performance
The extracted features were fed into SVM, speci cally a one-vs-all support vector classi er(SVC) (Cortes and Vapnik, 1995). This classi er is a common choice for these types ofapplications because it is memory ecient (uses only support vectors), and because it
works well with high dimensional spaces (Vapnik, 2000) and with unbalanced datasets (Heand Garcia, 2009). We validated our results using a tenfold strati ed random samplingstrategy without replacement. In each iteration, one subset was used as the test set, while
the classi er was trained on the remaining nine. As the evaluation metric we used accuracy
averaged across individual subsets. A similar validation strategy was utilized across otherexperiments in this thesis unless otherwise noted.

25

2.1.4

Results

Evaluation of individual steps

100

97

95

85

80

avg
D1
D2
D3

75

70

224x224320x320416x416

512x512

avg

norm
D1
D2
D3

60
50
c1

c2

c3

c4

LayersImage size

c5

95
94

c3

93

c4
92

c5
fused

91

D4

D4

70
128x128

max

96

80

Accuracy (%)

Accuracy (%)

90

Accuracy (%)

90

fused

global

2x2

4x4

7x7

14x14 28x28Size of preserved features

Figure 2.1: We show A) the e ect of the image size and pooling strategy (left); B) thee ect of the feature depth, normalization and feature fusion (center); and C) the e ect of
non-global pooling in one of our datasets - D3 (right). Adapted from paper I.

Impact of image size. The rst step in our experiments was to nd an appropriate
image size that would perform well across tasks and datasets. We focused on c5 features
and assessed the performance for several input sizes (Fig 2.1 - left). We found that theaccuracy increased until the size of 416x416 on most of the datasets, and that in somecases using even larger images resulted in worse performance. Thus, we decided to proceedwith 416x416 input image size.
Impact of pooling strategy. In our experiments, global average pooling yielded better
results than global max pooling. This could be explained by the fact that, in our datasets,
objects occupy a large portion of the image. Thus, by averaging, we allow more object
information to a ect the nal feature state than if we simply take a single value (the max),as in max pooling. Concatenating the two feature vectors obtained from the two di erent
pooling strategies yielded results that were intermediate between the two separate results.

26

Impact of feature depth. Our results show that c4 features perform the best, while c3
yields results that are comparable with those of c5.
Impact of feature normalization. Signed square root normalization increased theperformance but, somewhat surprisingly, we found that l2 -normalization had a negativee ect on accuracy, regardless of whether it was performed with or without signed squareroot normalization.
Impact of feature fusion. Feature fusion further improved the accuracy. The only
exception was on D3, where fusion marginally fell behind the single best layer (only one
image di erence in accuracy).
Impact of non-global pooling. In this experiment we obtained further improvementson three out of four datasets. For D1-D3, we found that pooling down to 4x4xF yielded
the best results, surpassing the globally pooled features. The most evident advantage ofan intermediate level of pooling was on D3. We did not see any improvements at all on
D4. This was the only dataset with non-centered and occasionally small objects, which
could potentially result in having many receptive elds with no information about theobject. In the three remaining datasets (D1-D3), the objects were always large, presumably
generating a signi cant number of receptive elds in later CNN layers.
Impact of image aspect ratio. In all experiments reported above, the image aspectratio was maintained. However, we were able to slightly improve the best performing modelfrom the previous experiment by resizing images without maintaining the aspect ratio.
Obviously, such resizing distorts objects but it also provides more information because
there are no added uninformative pixels.
Performance on related tasks.
To validate our conclusions, we evaluated our optimal solution on several recently publishedbiological image classi cation tasks (see Table 2.2). Our approach performed about as well
27

Table 2.2: Comparison of performance of our method on some recently published biologicalimage classi cation tasks. We used input images of size 416x416 (160x160 for Pollen23),
global average pooling, fusion of c1 {c5 features, and signed square root normalization.
Accuracy is reported using tenfold cross validation except for EcuadorMoths, where we used
the same partitioning as in the original study. Bold font indicates the best identi cation
performance. The result on EcuadorMoths in parenthesis is from a single c4 block. (fm =families; sp = species). Adapted from paper I.
Datasets

Classes

Our

OthersReference

MethodClearedLeaf

19 fm

88.7

71

Wilf et al. (2016)

SIFT + SVM

CRLeaves

255 sp

94.67

51

Carranza-Rojas et al. (2017)

netune InceptionV3

EcuadorMoths

675 sp55.4

55.7

Rodner et al. (2015)

AlexNet+SVM

EcuadorMoths

675 sp(58.2)

57.2

Wei et al. (2017)

VGG16+SCDA+SVM

Flavia

32 sp

99.95

99.6

Sun et al. (2017)

ResNet26

Pollen23

23 sp

94.8

64

Goncalves et al. (2016)

CST + BOW

as or better than previously published methods.

2.2

Paper II

2.2.1

Material and methods

For the experiments in paper II, we created two image datasets on 10 visually similar
species of ower chafer beetles of the genus Oxythyrea. For the rst dataset, we collected
images using a standardized taxonomic imaging setup, in which images with di erent depthof eld were stacked together in a single high resolution image. For the second dataset,the same specimens were photographed in a much simpler and faster way using only asmartphone and a cheap 2$ attachable lens (see Figure 2.2).
We rst experimented with images of di erent habitus views (dorsal and ventral). Hu-

28

cinctella

dulcisfunesta

gronbechi

noemi

pantherina subcalva tripolitana

both sexes

albopicta

B

dorsalmales

females

smartphone

abigail

femalesmales

ventralA

Figure 2.2: Fig. 2. Datasets of ten Oxythyrea species used in paper II. We
show example images of dataset B ( rst row) collected with a smartphone and a cheap
2$ attachable lens and example images from dataset A (remaining rows) collected in astandardized taxonomic imaging setting. Dataset A contains images of dorsal and ventral habitus including images of both sexes. Note that Oxythyrea beetles show sexual
dimorphism only on their abdomen (ventral view). Adapted from paper II.

29

mans often nd one habitus view more useful for identi cation than the other and weinvestigated whether this was also true for the ATIs we developed. Then, we investigated
how "quick and dirty" image collection using a smartphone and a cheap attachable lenscompares with the time-consuming standard taxonomic imaging setting. Lastly, we explored whether the same approach was applicable on tasks unsolvable by humans. Here,
we experimented with separating Oxythyrea specimens by sex using images of the dorsal
view. According to previous work on Oxythyrea, these species do not show any sexual
dimorphism in this view.
The experiments in paper II were based on optimal parameters from paper I. Specifically, we used VGG16 (Simonyan and Zisserman, 2014) pretrained on ImageNet (Russakovsky et al., 2015) for feature extraction and SVC (Cortes and Vapnik, 1995) for classi cation. Features from all ve convolutional blocks were reduced using global average
pooling, then concatenated and normalized using the signed square-root method. The only
di erence was the input image size, which was set to smaller size (224x224 ) in order tospeed up the experiments. The validation approach was the same as in paper I.
In addition to the o -the-shelf approach developed in paper I, we repeated the same
experiments using some current state-of-art techniques in image recognition. Speci cally,
we utilized a well known CNN SE-ResNext101-32x4 (Hu et al., 2018) and ne-tuned apublicly available checkpoint pre-trained on the ImageNet dataset (Russakovsky et al.,2015). This architecture is a variant of ResNets (He et al., 2016) (networks with residuals
modules) with many improvements added subsequently, such as "cardinality" (Xie et al.,2017) and a squeeze and excite block (Hu et al., 2018). The learning rate (lr ) was adjusted
using the one cycle policy ((Smith, 2018, 2015; Smith and Topin, 2017)), with a maximum
lr set to 0.0006. We set the batch size to 12 and back-propagated accumulated gradientson every two iterations for a total of 12 epochs. As our optimization strategy, we used
an adaptive learning rate optimization algorithm called Adam (Kingma and Ba, 2014)
with a momentum of 0.9. Regularization was done using i) a dropout (Srivastava et al.,2014) layer (0.5) inserted before the last layer; ii) label smoothing as our classi cation
loss function (Szegedy et al., 2016); and iii) augmentation (zooming,
30

ipping, shifting,brightness, lighting, contrast and mixup (Zhang et al., 2017)). Lastly, with Class Activation
Maps (CAM) (Zhou et al., 2016), we visualized so-called heat maps for relevant categoryspeci c regions of images. These heat maps light up the regions that are important for the
AI system in identifying an image as belonging to a particular category.

2.2.2

Results

O -the-shelf approach
Our results using the o -the-shelf system suggest that either one of the habitus views(dorsal or ventral) can be successfully utilized in identifying the ten species of Oxythyrea.
However, better accuracy is achieved with images of dorsal habitus compared to ventral(3x smaller error rate). This nding corresponds to human perception of the diculty ofthe task. Combining information from both views, we observed only slight improvements.
This is likely caused by the above-mentioned di erence in error rate between the views. If
we had combined information from similarly performing views, one would have expected agreater positive impact of the combination.
The images collected by a smartphone and a cheap attachable lens performed almost
as well as the high-resolution images. Although humansnd such images dicult touse for identi cation, clearly inferior to high-resolution images, they are apparently oftensucient for machine identi cation. A possible reason for this is that the images fed tocurrent AI systems for image classi cation are reduced in size (often to around 224x224
pixels). After reduction in size, the high-resolution taxonomic images are probably quite
similar to the smartphone ones, possibly explaining why they do not result in signi cantly
better identi cation performance.
The o -the-shelf feature transfer solution found sexual dimorphism of the dorsal habitus
in at least some Oxythyrea species. In two species, O. albopicta and O. noemi, the model
seemed to be able to identify most of the males correctly, while the sex identi cations forfemale specimens were comparable to guessing randomly.

31

Figure 2.3: Class Activation Maps for the specimens from the species/sex task (only selectspecimens are depicted). Adapted from paper II.
Beyond o -the-shelf solution
As expected, ne-tuning yielded better results than the o -the-shelf solution, drasticallyreducing the error rates (2-5x). This approach also allowed us to compute heat maps,
which made it possible to compare machine reasoning about the identi cation task to thereasoning of taxonomists. The heat maps showed that the model was often focusing on
the pronotum. In species considered easier to identify, this was the only region that was
highlighted (O. albopicta, O. cinctella), while on more dicult species (e.g.

O. dulcis,

O. noemi ) the model used information from a wider region including the whole elytra.
According to the heat maps, O. tripolitana was easily identi ed using information from thesmall part between the pronotum and the scutellum. This species has an accumulation ofsetae in this region, which until now has not been seen as a reliable morphological character
among taxonomists.
When the task was sorting to sex alongside species based on the dorsal habitus, themodel again focused on the pronotum and sometimes on the elytra. It is not clear what
exactly is the discriminating feature. Looking at the heat maps from the ventral side,32

for six species the model easily recognized the median region, where the pattern of white
dots was present only in males (Fig. 2.3). In the remaining species, this pattern was notpresent in either sex. However, the same region was still highlighted. The reason was likely
a sex-speci c grove present in the median part of the abdomen, which is clearly visible if
the abdomen is examined from the side and with appropriate illumination. However, thisgroove was impossible for humans to see in most of the images of the ventral habitus usedin the experiment.

2.3

Paper III

2.3.1

Material and methods

In paper III, we explored the utility of taxonomic or phylogenetic information in training
and evaluating CNNs for identi cation of biological species, with the aim of improving
these systems so that they make less catastrophic errors. Speci cally, we included thebiological information during the training by adjusting targets with label smoothing (LS)based on taxonomic information (taxonomic label smoothing - TLS) or distances between
species in a reference phylogeny (phylogenetic label smoothing - PLS). Similar to paper II
we used netunning from a pretrained checkpoint and we compared our approach against
two well established baselines: one-hot encoding and standard LS (Szegedy et al., 2016).
In one-hot encoding, categories are represented as binary vectors of length equal to thenumber of unique categories, with 1 for the correct target and 0 for the other categories. LS
is a weighted average of the one-hot encoded labels and the uniform probability distribution
over labels. In both scenarios, networks are optimized toward targets using a distribution,in which categories are equally distant from each other. In such a setting, a network isequally penalized for every misidenti cation it makes. With our approach, hierarchicalinformation based on biological relatedness is incorporated in the target encoding. This
results in the network being less penalized for misidentifying categories that are closely
related biologically, and more penalized for mixing up distant categories.

33

Our approach di ers from one-hot encoding and standard label smoothing onlyin the target encoding. We use a weighted
average of one-hot encoded labels and a
non-uniform distribution over labels representing hierarchical biological information about the categories based on taxonomic ranks, anagenetic distance, or clado- Figure 2.4: Systems optimized toward onegenetic distance. For taxonomic rank, we hot or LS (Szegedy et al., 2016) targets ascounted the number of shared taxonomic sume all categories are equally distant fromlevels (genus and family) between the cor- each other and hence all errors are penalized
rect target and each of the other cate- equally. We propose to smooth the targets
gories. For anagenetic-distance smoothing, using hierarchical biological relatedness inforwe used the branch lengths separating the mation (taxonomy or phylogeny) so that syscorrect target and each of the other cat- tems are penalized more for erroneous identiegories on the reference tree as the dis-cations that are farther away from the cor-tance measure. For cladogenetic-distance rect category.
smoothing, we instead used the number ofedges separating the categories on the reference tree as the distance measure. Lastly, we
normalized the resulting values by subtracting them from the maximum value (so thatclosely related categories had the highest values); and then we normalized the values, thatis, we divided the values with the sum over all categories so that the sum of the distribution
was equal to 1 as in one-hot labels. For all hierarchical smoothing methods, we explored
several mix-in proportions (smoothing values), , of hierarchical information to binary information ( 2 f0:025; 0:05; 0:1; 0:2; 0:4g) on two image data sets: 38,000 outdoor imagesof 93 species of snakes and 2,600 habitus images of 153 species of Lepidoptera (butter ies
and moths) images obtained from GBIF (2020).
In addition to accuracy used in papers I-II, here we used two more common evaluation metrics: Top-N-accuracy or topN - if the correct category is among the N highest
34

probabilities, we count the answer as correct (in this study N=3); and f1 score macro a weighted average of recall and precision, where recall is the ratio of correctly predicted
positive observations to all observations in the actual category, and precision is the ratio ofcorrectly predicted positive observations to the total predicted positive observations. All
three of these evaluation metrics assume that all errors are equally bad. For that reason wealso measured the accuracy at the genus and family levels, and for snake dataset we report
the accuracy of predicting a relevant biological trait, namely whether a snake species isvenomous or not.

2.3.2

Results

Firstly, we evaluated the experiments with standard evaluation metrics. On the

rst

dataset, one-hot encoding gave better results than LS on accuracy and top3-accuracy,
but slightly worse on f1 scores with macro averaging. Systems trained using phylogenetically informed targets (TLS or PLS) with small smoothing values (0:025  0:1)

gave slightly better results than both benchmarks, with exception of anagenetic-distancesmoothing, which performed on par with the benchmarks. On the second dataset, the LS
benchmark consistently gave better results than the one-hot benchmark. Results from experiments with TLS and PLS were consistently better than the one-hot benchmark. When
compared to the second benchmark, LS, the inclusion of hierarchical information (TLS or
PLS) with intermediate smoothing values (0:05  0:4) gave similar accuracy and f1

scores, while it often gave better top3.
Our phylogenetically or taxonomically informed approach performed better than bothbenchmarks on evaluation metrics that take into account the hierarchical information.
Speci cally, we found that TLS or PLS (based on anagenetic or cladogenetic distances),
across a range of di erent smoothing values, yielded better results on the accuracy ofidenti cations at the genus or family level, or the accuracy of predicting an importantbiological trait (a snake being venomous). Thus, even if the systems trained using TLS
or PLS made the same number of errors as the benchmark reference systems, the errors

35

tended to be less serious in that the misidenti ed categories involved organisms that weremore closely related to each other.

2.4

Paper IV

2.4.1

Design of the study. Material and methods

The aim of paper IV was to describe lessons learned during Swedish Ladybird Project2018 (SLP2018), a citizen-science project focused on collecting smartphone images in order to develop an ATI tool for Swedish species of ladybird beetles. The rst phase, thecitizen-science (CS) part, was organized in the summer of 2018. Initially, we aimed atschoolchildren (ages 6-16), but after the initial press release the project caught the attention of many local and national media and attracted a lot of interest from potential
participants. Therefore, we decided to extend the project to include the interested publicin general, and preschool kids (up to age 6) in particular. We o ered teachers to registerin advance to allow direct communication with the project team and to receive additional
support. The preregistered classes were also provided with an \experiment kit", which
included a guide for teachers, a macro-lens for mobile devices. and a recently publishedcomprehensive eld guide to Swedish ladybird species. In total, 700 experiment kits weredispensed among participants. The aim was to provide one kit per 15 participating kids, solarger classes or sets of classes received more than one kit. Contributions were submitted
through an app speci cally made for this project. For the identi cation of the ladybird
species in the collected images, we relied on experts. After completion of the project anevaluation survey was sent out the registered participants.
For comparison purposes, we downloaded all the images of Swedish ladybirds available
through the Global Biodiversity Information Facility (GBIF) GBIF (2020). The taxonomic
identi cations of the images provided by GBIF were taken at face value. GBIF images contributed in 2018 were used for the evaluation of ATIs ("GBIF2018"), while the remaining
images (collected before 2018; "GBIF training") were used as an additional training set

36

that we could compare to the image set collected through the SLP2018 project. Speci cally,
to evaluate the SLP2018 contribution, we trained networks on SLP2018, GBIF training
and SLP2018+GBIF training, and compared the identi cation performance of the resulting ATIs. We ne-tuned a well known and light architecture ResNet50 (He et al., 2015)
from a publicly available checkpoint pretrained on ImageNet (Russakovsky et al., 2015).
The netunning procedure resembled the one used in paper II but with di erent hyperparameters.
As our evaluation metrics we used accuracy as described in paper I and f1 score macro
as described in paper III. In addition to these common metrics, we used f1 score macroon subsets of species created based on the number of images per species on GBIF: with thetwo dominant species, Harmonia axyridis and Coccinella septempunctata; other common
species (ranked 3-10 in abundance); and the remaining species.

2.4.2

Results

Almost 400 teachers registered to participate in the project. According to the 24 replies
we received in the evaluation survey, many of them supervised more than one class. In
average, the number of students per registered teacher was 31.5 (range 16-67), or in totalaround 12,000 children. If we consider unregistered participants, which also include schools
and preschools, and expeditions for kids organized by amateur entomologists and natural
history museums, we estimate that the total number of children participating in the project
was around 15,000.
The registered teachers were instructed to spend 1-2 hours of e ort on eld work. The
survey revealed that most of the groups had searched ladybirds multiple times (up to 28
times) and they mostly invested up to 2h (80%) on the task on each eld trip. Perhaps
even more astonishing was that some groups, presumably preschool children, had spentorders of magnitude more time than expected (replies included \every day", \all the timethroughout the project period" and \many times").
In total, we received over 5,000 images for almost half of the Swedish ladybird species

37

Figure 2.5: Geographical distribution of the contributed images. Most of the images camefrom the most populated counties Stockholm, Skane and V•astra G•otaland (a). In (b) weshow locations where images are taken (each dot represents a single image) and in (c) weshow the normalized contribution per capita for each of the 21 counties (darker is more).
Adapted from paper IV.

38

(30/71). The rst images were submitted in early June, followed by the rst peak during
the rst three weeks of summer (weeks 26-28). The summer was calm until weeks 35-38,
which brought the biggest contributions.
The contributions were received from all of the 21 counties of Sweden. The mostpopulated counties|Stockholm, V•astra G•otaland and Sk
ane|contributed most of theimages. However, when accounting for population size, Gotland stood out with 7 times as
many contributions per capita as the country's average.
The majority of the insects from contributed photos (68%) were identi ed as Coccinella septempunctata. Similar patterns were found in the individual months, with C.
septumpunctata comprising between 60 and 80% of the monthly contributions. In addition
to C. septempunctata, only three more species were represented by more than 3% of thetotal number of images: Psyllobora vigintiduopunctata (7.7%), Harmonia axyridis (5.8%)
and Adalia bipunctata (5.4%).
The experiments designed to analyze the contribution of SLP2018 in the context ofdeveloping ATIs showed that the estimated value of the SLP2018 data heavily depended
on the choice of evaluation metric. The SLP2018 data, when compared to the GBIF data,
did not contribute signi cantly to improving the accuracy or top3 scores. These metrics
favor majority categories (such as C. septempunctata and H. axyridis in our study) andthese were already well represented in the GBIF dataset with 80% of all the ladybirds
images on GBIF belonging to one of these two species.
However, adding SLP2018 data to GBIF training notably improved our second metric,
F1 score with macro averaging. The improvement came from minority categories, and we
could see that by comparing F1 scores over three subsets of species. The scores for the
rst subset, comprising the two dominant species (C. septempunctata and H. axyridis),remained largely unchanged. The scores for the second subset, containing the remainderof the common species, were slightly but clearly improved, while the scores for the last
subset, containing the rare (minority) species, bene ted the most from the addition of the
SLP2018 images.

39

Chapter 3
Discussion
The scientists of today think deeply
instead of clearly. One must be saneto think clearly, but one can think
deeply and be quite insane.
Nikola Tesla

Our results from paper I show that there is considerable room for optimization ofcurrent DL techniques so that they generate better ATIs in settings that are typically encountered by taxonomists. Surprisingly, our experiments in paper I show that it is possible
to nd strategies that signi cantly boost the performance of ATI systems across a rangeof taxonomic tasks and datasets, and these strategies seem to be successful also on related
tasks. Speci cally, the results obtained in paper I are based on a computationally ecient approach, namely feature transfer from a single feed-forward pass through a publicly
available checkpoint of the VGG16 (Simonyan and Zisserman, 2014) system pretrained on
the ImageNet classi cation task (Russakovsky et al., 2015). When compared to a baseline
application of this approach, we were able to signi cantly improve the performance of theresulting ATI by introducing each of the following modi cations or additions: larger image
input size 416x416, global average pooling, fusion of features from all ve convolutional
blocks and signed square root normalization.

40

Our results also indicate that it may be possible to further improve identi cation performance by optimizing the dimensionality reduction. In our experiments, global pooling(1x1xF) was not always optimal; pooling to small but not minimal feature matrices (2x2xF
or 4x4xF) often resulted in better performance. Speci cally, intermediate pooling resulted
in signi cant improvements on three out of four datasets.
It turned out that intermediate-level features extracted from the c4 layer were themost useful ones for the taxonomic tasks we tackled in paper I. According to Zheng et al.
(2016), this is not the case for generic image classi cation tasks (e.g. Caltech-101 (Li Fei-Fei
et al., 2006), Caltech-256 (Grin et al., 2007), and PASCAL VOC07 (Everingham et al.,2015)), nor for popular ne-grained identi cation tasks (Bird-200-2011 (Wah et al., 2011)
and Flower-102 (Nilsback and Zisserman, 2008)), where the most discriminating layers are
the rst fully connected layer and c5, respectively. A possible reason for this discrepancy
could be that our images are more distant from the ImageNet dataset than any of theabove-mentioned image datasets, or that di erences between categories in our datasets areeven more subtle than those in the previous ne-grained identi cation tasks.
From the perspective of insect taxonomy, perhaps the most interesting insight frompaper I is that, with optimized feature transfer, it is possible to develop high-performance
ATIs for a wide range of tasks without expertise in image analysis or machine learning.
Developing ATIs with identi cation accuracy on par with or exceeding that of human experts clearly seems to be within reach for non-experts. Thus, many taxonomists shouldbe able themselves to leverage the latest advances in CNNs and deep learning to develop
high-performance ATIs for just about any classi cation task of interest in insect systematics.
In paper II, we examined the potential of the system with the optimal feature transfer
protocol identi ed in paper I in testing a set of new taxonomic hypotheses. With nofurther optimization, our o -the-shelf solution from paper I proved to be very useful intackling several challenging identi cation tasks, some of which were previously thought tobe unsolvable by humans. A particularly striking example is the identi cation of sex fromimages of the dorsal habitus of Oxythyrea beetles; sexual dimorphism in dorsal habitus has
41not been noted previously in these beetles.
It is perhaps not surprising that the amount of data is important for good performanceof machine identi cation (the more examples, the better; paper I) or that the dicultyincreases with the amount of morphological variation within categories (sexual dimorphism,
life stages, age, etc.; paper I, II). Thus, collecting a sucient number of accurately labeledimages for developing a suciently competent ATI is important, but may appear to bea dicult task, particularly since these image sets often have to be assembled essentiallyfrom scratch for many taxonomic tasks. Fortunately, our results (paper II) indicate that,at least for some tasks, images collected with a smartphone and a cheap attachable lens arequite sucient. Thus, photographing specimens with a smartphone sometimes o ers aninteresting alternative to the time-consuming procedure of collecting high resolution images
in a standardized taxonomic imaging setup. We estimated that by using a smartphone,
one may collect 40x more images in the same amount of time as that required for the
traditional setup. It is worth noting that an ATI based on a larger set of smartphone
images may actually perform better than a system trained on a smaller number of highresolution images. The smartphone approach is also much easier to scale as everyone hasa smartphone today, while the setup used for high-resolution imaging of insects is mostly
restricted to entomologists at larger institutions and requires a substantial investment inequipment.
In both paper I and II, we observed that machines and humans tend to nd the sameor similar tasks challenging. Good examples are separating visually similar species (sibling
species) (paper I, II) or placing a species to the correct higher taxonomic group (family
or tribe in our case) when the subgroup it belongs to (species and genus in our case) has
not been seen before (paper I). However, we found occasional cases where machines madecatastrophic errors that no human would make, something we tried to address in paper
III (see below).
An o -the-shelf solution can be sucient for developing an ATI with adequate performance for many tasks, as shown in papers I-II, but even when this is not the case, sucha solution is useful in testing whether an identi cation task is feasible at all, and whether42

it is worth spending more resources on developing a more sophisticated system. Going
beyond o -the-shelf and utilizing state-of-the-art techniques can improve AI performance,
but also give additional insights, as illustrated by our heat maps (class activation maps).
This technique highlights the regions of images containing the features that a model nds
informative for a given task (Fig. 2.3). Our experiments (paper II) indicate that humanexperts often nd features occurring in the highlighted regions useful for species discrimination. However, these heat maps are not always easy to interpret. They do not tell us
exactly what the discriminating features are but rather indicate where they are located.
For example, consider an image of a beetle specimen, and assume that a region of thespecimen with a lot of hairs is highlighted by the heat map (as in our experiments). This
does not tell us what feature of the hairs is important for discrimination (their number,
distribution, thickness, length or something else). In fact, we cannot even be sure thatthe discriminating feature has anything to do with the hairs, it could be some underlying
feature, such as the shape or size of the hairy body part. Therefore, we should not expect
the heat maps to point exactly to an interesting morphological feature but rather to give
us hints of where to look. Then, it is up to the expert to hypothesize what the actual
discriminating feature is.
Another interesting phenomenon that we were able to demonstrate with the heat maps
was the fact that CNNs learn from whole images, including information that may beirrelevant for the task. Speci cally, the heat maps from one of the species studied in
paper highlighted the region where the pin is located. It turned out that all the imaged
specimens of this species were collected by a single entomologist, who was pinning themfrom a non-standard angle, and this entomologist did not contribute any specimens of theother species. This is an excellent example of how bias can be introduced in image training
sets.
Intuitively, one might expect that more sophisticated training of AI systems could
reduce the risk of such irrelevant biases, or the risk of making catastrophic errors, thatis, misidentifying categories that are completely unrelated. In paper III we explore thisidea by proposing to utilize phylogenetic relationships among biological organisms (TLS
43and PLS) in the training of ATIs. Our solution is inspired by the recently introduced
LS (Szegedy et al., 2016) technique, which uses a weighted average of one-hot targets
and uniform distribution over labels during training. However, unlike both LS and onehot encoding, TLS and PLS use target encodings based on non-uniform distributions.
Speci cally, they mix one-hot encoding with a small proportion of a distribution basedon a hierarchical scoring scheme re ecting either taxonomic (TLS) or phylogenetic (PLS)
information. This rewards the system being trained for predicting the correct category,
while punishing it for errors in proportion to how much these errors violate what we know
about the biological relationships among the categories. The technique we propose could
easily be applied also in other cases where there is a natural hierarchy describing thesimilarity relations among the categories. As demonstrated by our results, systems trainedin this way tend to make less serious errors. For this reason, hierarchically aware systems
might be preferred in many practical applications over standard systems, even when they
make more errors in total. Examples may include cases where the cost of making an error
is very high, such as misidentifying a venomous snake for a non-venomous one.
The results presented in paper III indicate that systems trained with phylogenetic
information often perform on par with or better than baseline systems in terms of common
evaluation metrics, such as accuracy, topK-accuracy and f1 score macro. When evaluatedwith custom metrics that take the biological context into account (accuracy on the genus
and family levels, or accuracy in predicting a biological trait) we observed that the systems trained using TLS or PLS often outperformed both of the baseline systems. In our
experiments, a range of smoothing ratios ( values) was tested, and although 0.05 and 0.1
seemed to perform well on both datasets, we noticed that applying the same smoothingratio across tasks might be suboptimal given the current methodological setup. The reason
for this is possibly the di erence in the number of categories across tasks and the fact that
we distribute total smoothing value

across all categories. As a result, tasks with a larger

number of categories have smaller average smoothing value per category. A possible solution could be to distribute the smoothing only across a xed number of the most related
categories and keep the remaining categories at zero in the smoothing distribution.
44

Assembling suciently large training sets of accurately labeled images will remain achallenge in the development of ATIs for many organism groups. Using citizen-science
contributions is an obvious possibility, and our attempt in this direction (paper IV)
exceeded all expectations. We estimated that approximately 15,000 children participatedin the project. Given our conservative estimate that each child devoted 1 hour to the task,
this translates to a contribution of approximately 15,000 hours (375 weeks or 83 person
months) of e ort to the project. If a single person would be required to perform this task
by himself/herself, it would take that person more than 10 years to complete it ( 1,600
annual working hours in Sweden (Charlie Giattino and Roser, 2013)).
Over 5,000 images were received in two major waves, one in the early summer and one
in the late summer. These waves were anticipated as a result of the design of the project,which closely followed the biology of ladybirds in Sweden. The adult ladybirds become
active in May-June searching for food and to mate, then they slow down until the end of thesummer when the new generation emerges, and joins the adult beetles who overwinteredin the search for food to stack reserves for the upcoming winter. The natural life-history
uctuation in adult beetle activity were probably further accentuated by the timing of
project activities. The rst wave coincided with the project kick-o and the launch of an
extensive project marketing campaign. We did have participants that contributed images
during the whole summer (preschools,

eld stations, museums and groups of amateur

entomologists), but the image contributions were fewer during this period. Finally, themajority of the contributed images were received in the second wave in the late summer,
when schoolchildren returned to school and teachers had scheduled project activities in theeld.
The results were impressive in terms of both species coverage and taxonomic composition. In total, 30 of the 71 naturally occurring coccinellid species were photographed.
The most common species were photographed the most, while some rare species were completely absent from the contributed images. An interesting nding was that Harmonia
axyridis, an invasive species that was rst recorded from Sweden a decade ago (Brown
et al., 2007), was the third most commonly photographed species. As one might expect,45we noted some biases toward species that thrive in urban environments and against thosethat occur in habitats less suitable for eld excursions with children or in less populatedparts of the country (northern species).
The 5,119 images collected by the SLP2018 project represent a signi cant expansionof what was at the time available through GBIF. SLP2018 images increased the number
of Swedish images of ladybird beetles by a factor of 50. The number of SLP2018 images
approached the number of images submitted that year to GBIF from around the world(7,264). and represent 21% of all GBIF images collected over the last 20 years from allover the globe of species of ladybird beetles occurring in Sweden. These numbers clearlyshow the potential of citizen-science projects in quickly assembling sizeable sets of images
for suitable organism groups.
Our experiments also con rm that the SLP2018 data provided valuable information
for training ATIs. The increase in identi cation accuracy was particularly obvious for theless common species, the ones for which the number of images available previously wasinsucient for adequate training of ATIs. It is interesting to speculate on how useful
citizen-science projects might be in general for assembling suitable image sets for training
ATIs. Clearly, the most signi cant boosts will be expected for organism groups and species
for which there are few existing images. The group also needs to be fairly easy to identify
and to photograph for the participating citizen scientists. We think that a fair numberof groups of insects, other invertebrates, fungi and plants mayt these criteria. For

some of the more obscure groups, it may sometimes be possible to nd more specialized
amateur naturalists that are willing to contribute, while other groups are suitable for largerscale projects targeting non-specialists as in the SLP2018 project. An interesting lesson is
the outsize contribution of the images of the rare species and of the rarely photographedsubgroups, such as some of the subfamilies of Coccinellidae that were poorly represented inboth the SLP2018 and GBIF image sets. It may well be possible to increase the e ectivenessof citizen-science projects by directing the attention of participants towards these subgroups
or species, perhaps by considering various point reward systems. Clearly, citizen science
provides many valuable opportunities for advancing the development of ATIs. At the same46

time, these projects can also help raise the awareness of the value of biodiversity and have
many other positive societal side e ects.

47

Chapter 4
Concluding remarks and thoughts on
not so far future
The present is theirs; the future, forwhich I really worked, is mine.
Nikola Tesla

The research presented in this thesis demonstrates how current AI technologies |
speci cally CNNs and DL | can solve several types of challenging taxonomic identi cation
tasks (papers I-II). We managed to develop an optimized feature extraction protocol thatwas successful in tackling a broad range of taxonomic identi cation tasks, making it possible
to o er an easy-to-use instant-return approach for taxonomists interested in investigating
the feasibility of using AI in addressing particular identi cation problems. Speci cally, wedemonstrated the usefulness of our optimized solution in: (1) identifying insects to higher
groups when they are likely to belong to subgroups that have not been seen previously; (2)identifying visually similar species that are dicult to separate even for experts; and (3)solving some identi cation tasks that appear unsolvable to humans, such as detecting thesex of a specimen when there is no prior evidence in the literature of sexual dimorphism.
We also showed that going beyond such an o -the-shelf solution by utilizing stateof-the-art AI techniques can yield further improvements in identi cation accuracy while48

providing additional insights. A particularly striking example of the latter is provided by
the heat maps presented in paper II, highlighting regions that are particularly important
for the ATI in discriminating among categories of imaged objects. This type of approach
may well turn out to be a powerful tool that can guide experts in identifying the informative
morphological features that are critical for both taxonomic and phylogenetic research.
In paper III, we demonstrated that by leveraging taxonomic or phylogenetic information we can train systems that are as good as standard systems in terms of common
evaluation metrics (accuracy, topK accuracy, f1 score macro), but make errors that areless wrong in an objective biological sense (paper III). An obvious research direction inspired by this paper would be to explore the use of CNNs for phylogenetic inference. My
initial experiments show that models optimized only toward targets based on phylogenetic
information are quite good at approximating the position of species in the tree of life. Of
course, this requires that a reference phylogeny is available for training. However, there are
numerous ways in which such phylogeny-trained AI systems could be useful. For instance,a system trained using an approximate or incomplete reference phylogeny could learn todistinguish phylogenetically important features, and thus re ne the phylogeny in the way
it perceives similarities among images. It should also be able to place unseen species in aphylogenetic context more accurately than other systems. It may also turn out that AI\knowledge" about what image features structure the phylogeny of one group of organisms
can transfer to related groups. If so, then an AI system trained to reconstruct phylogeny
may well be able to infer phylogeny of a group of organisms it has not seen previously
using only images of those organisms.
We already mentioned that the credit for recent developments in computer vision goes tothe faster and cheaper computational power, improved algorithms, and increasing amountsof image training data available on the internet. We are witnessing constant, fast-paced
improvements in the rst two, regardless of what happens in the taxonomic community.
However, the taxonomic community could contribute substantially to the eld making evenbigger strides forward by generating sucient amounts of accurately labeled image data fortraining of AI systems. An obvious way to generate more data is to use CS e orts similar to49

our project described in paper IV. However, the CS path does not seem a viable solution
to the problem of accumulating adequate training sets for a signi cation number of insect
species. Some organism groups are simply too dicult for citizen scientists to nd in theeld, or to photograph in such a way that it will be possible to identify the imaged species.
In some cases one may be able to extend the limits of what can be done by focusing on
specialized amateur naturalists, to train amateur naturalists to work with new groups, orto elicit the help of CS in imaging existing collections of specimens. However, in the end,
accumulating sucient training data has to be achieved using other approaches for manyorganism groups.
An interesting alternative approach to CS would be to use camera traps for accumulating images for training. Camera traps have the additional advantage that they can also
be used for continuous monitoring. There is already considerable experience with camera
traps for mammals, and with developing AI techniques for these data. For example, in
the Snapshot Serengeti project (Swanson et al., 2015), CNN-based models have now beendeveloped that are better than humans in terms of accuracy at recognizing common species(zebra, lion, elephant)(Norouzzadeh et al., 2018; Willi et al., 2019). There are also several
successful examples targeting insects, mainly driven by industrial applications (e.g. see therecent review paper on automated monitoring of pests (Cardim Ferreira Lima et al., 2020)
and references therein). However, as in the CS approach, we need to nd ways how to generate labels for the images we collect. For instance, by the year 2015, Snapshot Serengeti
had accumulated over 1,500,000 labeled images by using an army of 100,000 volunteers.
In this project, it was found that having an image labeled by one volunteer gave only 85%
accuracy in the identi cation. This improved further to 95% and 98% accuracy when using
the voting consensus of 10 and 20 volunteers, respectively (see Swanson et al. (2015) for
details of the setting). A single expert could label images with 96.6% accuracy, CNNs
alone reached 97% accuracy, and CNN+expert in a human-in-the-loop approach reached99.8% accuracy. Unfortunately, insect identi cation is considerably more challenging forhumans than mammal identi cation for several reasons, including the enormous diversityof insects, and becoming an expert on even a single group can require a lifetime of training.
50

Thus, entomologists need to be more creative than mammalogists in order to get over thishump caused by the need for sizeable collections of accurately labeled training data.
Most of the challenges we nd in building AI identi cation systems for taxonomy are
not unique to our eld. In fact, we can count on constant advancements in DL and computer vision to also bene t the construction of ATIs. For instance, there has recently beensigni cant progress in utilizing unlabeled images in training (Chen et al., 2020; He et al.,2020), which considerably reduces the need for labeled data at the expense of computational resources. Because computational resources are getting cheaper and faster, while
human labor costs for labeling are increasing, this seems a particularly promising future
direction for the development of ATIs and many other AI applications. Another researcharea of potential interest to taxonomists, where the DL community is very active, is socalled automated machine learning (AutoML) (see (He et al., 2021) and references therein).
The idea is to build solutions based on cutting-edge techniques but without any human
involvement, thus completely removing the need for machine learning expertise. In otherwords, the idea is to completely automate data preparation, feature engineering, hyperparameter optimization, and neural architecture search. There are many other promisingideas currently pursued by the DL community, which will, sooner or later, a ect many
aspect of taxonomy and transform the way we work with species discovery, description andidenti cation.
My impression is that the time is ripe for biologists experimenting with AI techniques to
set bigger goals that could provide more substantial bene ts and bring more transformativechanges. For instance, imagine a robot with a camera, which could capture specimens,
automatically detect them and recognize most of the species, collect behavioural data, andthen, depending on a prede ned set of rules, select specimens to be collected and preserved
for future study, eliminating specimens of invasive or harmful species, and releasing theremainder. How about nano-robots small enough so that millions could t on a single
head of an insect pin; if they can ow through our bloodstreams, detect and cure diseases
then it might be possible to have similar nano-robots y around in the wild, detect aninteresting specimen, collect behavioural data and sample tissue for genetic analysis in51

a non-destructive way. The recent progress in technology gives me con dence that thesetypes of systems might be feasible in the near future, and I hope I will be able to contributeto the journey towards building them.
In conclusion, today's "machines" can see, sometimes better than us, they learn much
faster, they never forget and their knowledge is transferable to new tasks in similar domains.
Technology enables us already today to automatize many aspects of taxonomic work, and
it is only a matter of time until we will be able to put together some pieces of technology
already perfected in various other industries to achieve even more impressive tasks. Taking
advantage of these opportunities will enable us to progress more forcefully than ever toward
the goal of saving the Earth. Despite all the challenges, I think the future is bright, as
those challenges are nothing but new opportunities.

52

Bibliography
Agnarsson,I. and Kuntner,M. Taxonomy in a changing world: seeking solutions for ascience in crisis. Systematic Biology, 56(3):531{539, 2007.
Arandjelovic,R. and Zisserman,A. All about VLAD. In Computer Vision and Pattern
Recognition (CVPR), 2013 IEEE Conference on, pages 1578{1585. cv-foundation.org,
2013.
Arbuckle,T., Schr•oder,S., Steinhage,V., and Wittmann,D. Biodiversity informatics in action: identi cation and monitoring of bee species using ABIS. In Proc. 15th Int. Symp.
Informatics for Environmental Protection, volume 1, pages 425{430. enviroinfo.eu, 2001.
Azizpour,H., Razavian,A. S., Sullivan,J., Maki,A., and Carlsson,S. Factors of transferability for a generic ConvNet representation. IEEE Trans. Pattern Anal. Mach. Intell., 38(9):1790{1802, September 2016.
Bengio,Y. Deep learning of representations for unsupervised and transfer learning. In
Proceedings of ICML workshop on unsupervised and transfer learning, pages 17{36, 2012.
Bottou,L., Curtis,F. E., and Nocedal,J. Optimization methods for large-scale machinelearning. Siam Review, 60(2):223{311, 2018.
Breiman,L. Random forests. Machine learning, 45(1):5{32, 2001.
Brown,P., Adriaens,T., Bathon,H., Cuppen,J., Goldarazena,A., H•agg,T., Kenis,M., Klausnitzer,B., Kovar,I., Loomans,A., et al. Harmonia axyridis in europe: spread and distri-53

bution of a non-native coccinellid. In From Biological Control to Invasion: the Ladybird
Harmonia axyridis as a Model Species, pages 5{21. Springer, 2007.
Cardim Ferreira Lima,M., Damascena de Almeida Leandro,M. E., Valero,C., Pereira Coronel,L. C., and Goncalves Bazzo,C. O. Automatic detection and monitoring of insect
pests|a review. Agriculture, 10(5):161, 2020.
Carranza-Rojas,J., Goeau,H., Bonnet,P., Mata-Montero,E., and Joly,A. Going deeper
in the automated identi cation of herbarium specimens. BMC Evol. Biol., 17(1):181,
August 2017.
Caruana,R. Learning many related tasks at the same time with backpropagation. In
Advances in neural information processing systems, pages 657{664, 1995.
Ceballos,G., Ehrlich,P. R., and Dirzo,R. Biological annihilation via the ongoing sixth
mass extinction signaled by vertebrate population losses and declines. Proceedings of thenational academy of sciences, 114(30):E6089{E6096, 2017.
Chapman,A. D. et al. Numbers of living species in australia and the world. 2009.
Charlie Giattino,E. O.-O. and Roser,M. Working hours. Our World in Data, 2013.
https://ourworldindata.org/working-hours.
Chen,T., Kornblith,S., Norouzi,M., and Hinton,G. A simple framework for contrastive
learning of visual representations. arXiv preprint arXiv:2002.05709, 2020.
Ciresan,D., Meier,U., Masci,J., and Schmidhuber,J. A committee of neural networks fortrac sign classi cation. In The 2011 international joint conference on neural networks,
pages 1918{1921. IEEE, 2011.
Coleman,C. O. Taxonomy in times of the taxonomic impediment{examples from the community of experts on amphipod crustaceans. Journal of Crustacean Biology, 35(6):729{740, 2015.

54

Cortes,C. and Vapnik,V. Support-vector networks. Mach. Learn., 20(3):273{297, September 1995.
Cox,D. R. The regression analysis of binary sequences. Journal of the Royal StatisticalSociety: Series B (Methodological), 20(2):215{232, 1958.
Cui,Y., Song,Y., Sun,C., Howard,A., and Belongie,S. Large scale ne-grained categorization and domain-speci c transfer learning. In Proceedings of the IEEE Conference on
Computer Vision and Pattern Recognition (CVPR), June 2018.
Dirzo,R., Young,H. S., Galetti,M., Ceballos,G., Isaac,N. J., and Collen,B. Defaunation in
the anthropocene. science, 345(6195):401{406, 2014.
Donahue,J., Jia,Y., Vinyals,O., Ho man,J., Zhang,N., Tzeng,E., and Darrell,T. DeCAF:
A deep convolutional activation feature for generic visual recognition. In International
Conference on Machine Learning, pages 647{655. jmlr.org, January 2014.
Ebach,M. C., Valdecasas,A. G., and Wheeler,Q. D. Impediments to taxonomy and usersof taxonomy: accessibility and impact evaluation. Cladistics, 27(5):550{557, 2011.
Ehrlich,P. R. The scale of human enterprise and biodiversity loss. Extinction rates, 1995.
Everingham,M., Eslami,S. M. A., Van-Gool,L., Williams,C. K. I., Winn,J., and Zisserman,A. The Pascal Visual Object Classes Challenge: A Retrospective. International
Journal of Computer Vision, 111(1):98{136, 2015.
Feng,L., Bhanu,B., and Heraty,J. A software system for automated identi cation andretrieval of moth images based on wing attributes. Pattern Recognit., 51:225{241, March
2016.
Francoy,T. M., Wittmann,D., Drauschke,M., M•
uller,S., Steinhage,V., Bezerra-Laure,M.
A. F., De Jong,D., and Goncalves,L. S. Identi cation of africanized honey bees throughwing morphometrics: two fast and ecient procedures.
September 2008.
55

Apidologie, 39(5):488{494,Friedman,J. H. Greedy function approximation: a gradient boosting machine. Annals of
statistics, pages 1189{1232, 2001.
Fukushima,K. Neocognitron: a self organizing neural network model for a mechanism ofpattern recognition una ected by shift in position. Biological cybernetics, 36(4):193{202,1980. URL http://www.ncbi.nlm.nih.gov/pubmed/7370364.
Fukushima,K. Neural network model for a mechanism of pattern recognition una ectedby shift in position-neocognitron. IEICE Technical Report, A, 62(10):658{665, 1979.
Fukushima,K., Miyake,S., and Ito,T. Neocognitron: A neural network model for a mechanism of visual pattern recognition. IEEE Transactions on Systems, Man, and Cybernetics, SMC-13(5):826{834, sep 1983. ISSN 0018-9472. doi: 10.1109/TSMC.1983.6313076.
URL http://ieeexplore.ieee.org/document/6313076/.
Gaston,K. J. and O'Neill,M. A. Automated species identi cation: why not?

Philo-

sophical Transactions of the Royal Society B: Biological Sciences, 359(1444):655{
667, apr 2004. ISSN 0962-8436. doi: 10.1098/rstb.2003.1442. URL http://rstb.
royalsocietypublishing.org/content/359/1444/655.
Gaston,K. J. and May,R. M. Taxonomy of taxonomists. Nature, 356(6367):281{282, mar
1992. ISSN 0028-0836. doi: 10.1038/356281a0. URL http://dx.doi.org/10.1038/
356281a0.
Gauld,I. D., O'Neill,M. A., Gaston,K. J., and Others. Driving miss daisy: the performanceof an automated insect identi cation system. Hymenoptera: evolution, biodiversity andbiological control, pages 303{312, 2000.
GBIF. Home page. Available from: https://www.gbif.org, 2020. Accessed: November
2020.
Goncalves,A. B., Souza,J. S., Silva,G. G. d., Cereda,M. P., Pott,A., Naka,M. H., andPistori,H. Feature extraction and machine learning for the classi cation of braziliansavannah pollen grains. PLoS One, 11(6):e0157044, June 2016.
56

Grin,G., Holub,A., and Perona,P. Caltech-256 Object Category Dataset. Technicalreport, 2007.
Hansen,O. L., Svenning,J.-C., Olsen,K., Dupont,S., Garner,B. H., Iosi dis,A., Price,B. W.,
and Hye,T. T. Species-level image classi cation with convolutional neural network
enables insect identi cation from habitus images. Ecology and evolution, 10(2):737{747,2020.
Hanson,S. J. A stochastic version of the delta rule. Physica D: Nonlinear Phenomena, 42(1-3):265{272, 1990.
He,H. and Garcia,E. A. Learning from imbalanced data. IEEE Transactions on knowledge
and data engineering, 21(9):1263{1284, 2009.
He,K., Zhang,X., Ren,S., and Sun,J. Spatial pyramid pooling in deep convolutional networks for visual recognition. IEEE Trans. Pattern Anal. Mach. Intell., 37(9):1904{1916,
September 2015.
He,K., Zhang,X., Ren,S., and Sun,J. Deep residual learning for image recognition. In
Proceedings of the IEEE conference on computer vision and pattern recognition, pages
770{778, 2016.
He,K., Fan,H., Wu,Y., Xie,S., and Girshick,R. Momentum contrast for unsupervised visual
representation learning. In Proceedings of the IEEE/CVF Conference on Computer
Vision and Pattern Recognition (CVPR), June 2020.
He,X., Zhao,K., and Chu,X.

Automl: A survey of the state-of-the-art.

Based Systems, 212:106622, 2021.

ISSN 0950-7051.

Knowledge-

doi: https://doi.org/10.1016/

j.knosys.2020.106622. URL http://www.sciencedirect.com/science/article/pii/
S0950705120307516.
Hinton,G., Deng,L., Yu,D., Dahl,G., Mohamed,A.-r., Jaitly,N., Senior,A., Vanhoucke,V.,
Nguyen,P., Sainath,T., and Kingsbury,B. Deep Neural Networks for Acoustic Modeling in Speech Recognition: The Shared Views of Four Research Groups. IEEE Signal
57

Processing Magazine, 29(6):82{97, nov 2012. ISSN 1053-5888. doi: 10.1109/MSP.2012.
2205597. URL http://ieeexplore.ieee.org/document/6296526/.
Hu,J., Shen,L., and Sun,G. Squeeze-and-excitation networks. In The IEEE Conference on
Computer Vision and Pattern Recognition (CVPR), June 2018.
Io e,S. and Szegedy,C. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03167, 2015.
Jean,S., Cho,K., and Memisevic,R. On Using Very Large Target Vocabulary for Neural
Machine Translation. In ACL-IJCNLP, pages 1{10, 2015.
Joly,A., Go•eau,H., Botella,C., Glotin,H., Bonnet,P., Vellinga,W.-P., Planque,R., andM•uller,H. Overview of lifeclef 2018: a large-scale evaluation of species identi cation
and recommendation algorithms in the era of ai. In International Conference of the
Cross-Language Evaluation Forum for European Languages, pages 247{266. Springer,
2018.
Karpathy,A. and Fei-Fei,L. Deep Visual-Semantic Alignments for Generating Image Descriptions.

In The IEEE Conference on Computer Vision and Pattern Recognition(CVPR), pages 3128{3137, 2015.
Kelley,H. J. Gradient theory of optimal ight paths. Ars Journal, 30(10):947{954, 1960.
Kiefer,J., Wolfowitz,J., et al. Stochastic estimation of the maximum of a regression function. The Annals of Mathematical Statistics, 23(3):462{466, 1952.
Kingma,D. P. and Ba,J. Adam: A method for stochastic optimization. arXiv preprint
arXiv:1412.6980, 2014.
Krizhevsky,A., Sutskever,I., and Hinton,G. E. ImageNet Classi cation with Deep Convolutional Neural Networks. Advances In Neural Information Processing Systems, pages
1{9, 2012. ISSN 10495258.

58

Kumar,A.,

Irsoy,O.,

Ondruska,P.,

Iyyer,M.,

Bradbury,J.,

Gulrajani,I.,

Zhong,V.,

Paulus,R., and Socher,R. Ask Me Anything: Dynamic Memory Networks for Natural
Language Processing. jun 2016. URL http://arxiv.org/abs/1506.07285.
Laliberte,A. S. and Ripple,W. J. Range contractions of north american carnivores andungulates. BioScience, 54(2):123{138, 2004.
LeCun,Y., Boser,B., Denker,J. S., Henderson,D., Howard,R. E., Hubbard,W., andJackel,L. D. Backpropagation Applied to Handwritten Zip Code Recognition. NeuralComputation, 1(4):541{551, dec 1989. ISSN 0899-7667. doi: 10.1162/neco.1989.1.4.541.
URL http://www.mitpressjournals.org/doi/abs/10.1162/neco.1989.1.4.541.
LeCun,Y., Bengio,Y., and Hinton,G. Deep learning. nature, 521(7553):436{444, 2015.
Li Fei-Fei, Fergus,R., and Perona,P. One-shot learning of object categories. In IEEE Transactions on Pattern Analysis and Machine Intelligence, volume 28, pages 594{611, apr
2006. doi: 10.1109/TPAMI.2006.79. URL http://ieeexplore.ieee.org/document/
1597116/.
Lin,M., Chen,Q., and Yan,S. Network in network. December 2013.
Linnainmaa,S. Taylor expansion of the accumulated rounding error. BIT Numerical Mathematics, 16(2):146{160, 1976.
Lytle,D. A., Martnez-Mu~
noz,G., Zhang,W., Larios,N., Shapiro,L., Paasch,R., Moldenke,A., Mortensen,E. N., Todorovic,S., and Dietterich,T. G. Automated processing
and identi cation of benthic invertebrate samples. J. North Am. Benthol. Soc., 29(3):867{874, June 2010.
Marcus,L.
MichiganF.

Traditional

morphometrics,

morphometrics.

2:77{122,

1990.

Proceedings
URL

of

the

https://www.

researchgate.net/profile/F{%}7B{_}{%}7DRohlf/publication/

30850286{%}7B{_}{%}7DProceedings{%}7B{_}{%}7Dof{%}7B{_}{%}7Dthe{%}7B{_}{%}7DMichigan{%}7B{
links/5566227508aefcb861d1971b.pdf{%}7B{#}{%}7Dpage=87.
59

Martineau,M., Conte,D., Raveaux,R., Arnault,I., Munier,D., and Venturini,G. A survey
on image-based insects classi cation. Pattern Recognit., 2016.
Maxwell,S. L., Fuller,R. A., Brooks,T. M., and Watson,J. E. Biodiversity: The ravages ofguns, nets and bulldozers. Nature News, 536(7615):143, 2016.
Mora,C., Tittensor,D. P., Adl,S., Simpson,A. G., and Worm,B. How many species are
there on earth and in the ocean? PLoS biology, 9(8):e1001127, 2011.
Nilsback,M.-E. and Zisserman,A. Automated ower classi cation over a large number ofclasses. In Indian Conference on Computer Vision, Graphics and Image Processing, Dec
2008.
Norouzzadeh,M. S., Nguyen,A., Kosmala,M., Swanson,A., Palmer,M. S., Packer,C., andClune,J. Automatically identifying, counting, and describing wild animals in cameratrap images with deep learning. Proceedings of the National Academy of Sciences, 115(25):E5716{E5725, 2018.
Novotny,V., Basset,Y., Miller,S. E., Weiblen,G. D., Bremer,B., Cizek,L., and Drozd,P. Low
host speci city of herbivorous insects in a tropical forest. Nature, 416(6883):841, 2002.
O'Neill,M. A. DAISY: a practical tool for semi-automated species identi cation. Automated taxon identi cation in systematics: theory, approaches, and applications. CRC
Press/Taylor & Francis Group, Boca Raton/Florida, pages 101{114, 2007.
Oquab,M., Bottou,L., Laptev,I., and Sivic,J. Learning and transferring mid-level image
representations using convolutional neural networks. Proc. IEEE, 2014.
Ripple,W. J., Estes,J. A., Beschta,R. L., Wilmers,C. C., Ritchie,E. G., Hebblewhite,M.,
Berger,J., Elmhagen,B., Letnic,M., Nelson,M. P., et al. Status and ecological e ects of
the world's largest carnivores. Science, 343(6167), 2014.
Robbins,H. and Monro,S. A stochastic approximation method. The annals of mathematicalstatistics, pages 400{407, 1951.
60

Rodman,J. E. and Cody,J. H. The taxonomic impediment overcome: Nsf's partnerships forenhancing expertise in taxonomy (peet) as a model. Systematic biology, 52(3):428{435,2003.
Rodner,E., Simon,M., Brehm,G., Pietsch,S., Wolfgang W•agele,J., and Denzler,J. Finegrained recognition datasets for biodiversity analysis. July 2015.
Rohlf,J. F. and Marcus,L. F.

A revolution morphometrics.

ogy & evolution, 8(4):129{32, apr 1993.
0169-5347(93)90024-J.

URL

Trends in ecol-

ISSN 0169-5347.

doi:

10.1016/

http://www.sciencedirect.com/science/

article/pii/016953479390024Jhttps://www.researchgate.net/publication/
49756524{%}7B{_}{%}7DA{%}7B{_}{%}7DRevolution{%}7B{_}{%}7Din{%}7B{_}{%}7DMorphometrics.
Rosenblatt,F. The perceptron, a perceiving and recognizing automaton Project Para. Cornell
Aeronautical Laboratory, 1957.
Rumelhart,D. E., Hinton,G. E., and Williams,R. J. Learning representations by backpropagating errors. nature, 323(6088):533{536, 1986.
Russakovsky,O., Deng,J., Su,H., Krause,J., Satheesh,S., Ma,S., Huang,Z., Karpathy,A.,
Khosla,A., Bernstein,M., Berg,A. C., and Fei-Fei,L. ImageNet large scale visual recognition challenge. Int. J. Comput. Vis., 115(3):211{252, December 2015.
Sainath,T. N., Mohamed,A.-r., Kingsbury,B., and Ramabhadran,B. Deep convolutional
neural networks for LVCSR. In 2013 IEEE International Conference on Acoustics, Speechand Signal Processing, pages 8614{8618. IEEE, may 2013. ISBN 978-1-4799-0356-6.
doi: 10.1109/ICASSP.2013.6639347. URL http://ieeexplore.ieee.org/document/
6639347/.
Schmidhuber,J. Who invented backpropagation? More in [DL2], 2014.
Schmidhuber,J. Deep learning in neural networks: An overview. Neural networks, 61:
85{117, 2015.
61

Schr•oder,S., Drescher,W., Steinhage,V., and Kastenholz,B. An automated method forthe identi cation of bee species (hymenoptera: Apoidea). In Proc. Intern. Symp. on
Conserving Europe's Bees. Int. Bee Research Ass. & Linnean Society, London, pages
6{7, 1995.
Sharif Razavian,A., Azizpour,H., Sullivan,J., and Carlsson,S. CNN features o -the-shelf:
an astounding baseline for recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pages 806{813. cv-foundation.org, 2014.
Simonyan,K. and Zisserman,A. Very deep convolutional networks for Large-Scale image
recognition. September 2014.
Smith,L. N. No more pesky learning rate guessing games. CoRR, abs/1506.01186, 2015.
URL http://arxiv.org/abs/1506.01186.
Smith,L. N. A disciplined approach to neural network hyper-parameters: Part 1{learning
rate, batch size, momentum, and weight decay. arXiv preprint arXiv:1803.09820, 2018.
Smith,L. N. and Topin,N. Super-convergence: Very fast training of residual networks usinglarge learning rates. CoRR, abs/1708.07120, 2017. URL http://arxiv.org/abs/1708.
07120.
Srivastava,N., Hinton,G., Krizhevsky,A., Sutskever,I., and Salakhutdinov,R. Dropout: asimple way to prevent neural networks from over tting. The journal of machine learning
research, 15(1):1929{1958, 2014.
Stallkamp,J., Schlipsing,M., Salmen,J., and Igel,C. The german trac sign recognition
benchmark: a multi-class classi cation competition. In The 2011 international joint
conference on neural networks, pages 1453{1460. IEEE, 2011.
Steinhage,V., Schr•oder,S., Cremers,A. B., and Lampe,K.-H. Automated extraction andanalysis of morphological features for species identi cation. In Automated taxon identi cation in systematics: Theory, approaches and applications, pages 115{129. CRC Press,
2007.
62

Stork,N. E., McBroom,J., Gely,C., and Hamilton,A. J. New approaches narrow global
species estimates for beetles, insects, and terrestrial arthropods. Proceedings of the
National Academy of Sciences, 112(24):7519{7523, 2015.
Sun,Y., Liu,Y., Wang,G., and Zhang,H. Deep learning for plant identi cation in naturalenvironment. Comput. Intell. Neurosci., 2017:7361042, May 2017.
Sutskever,I., Vinyals,O., and Le,Q. V. Sequence to Sequence Learning with Neural Networks. In Advances in Neural Information Processing Systems, pages 3104{3112, 2014.
Swanson,A., Kosmala,M., Lintott,C., Simpson,R., Smith,A., and Packer,C.

Snapshotserengeti, high-frequency annotated camera trap images of 40 mammalian species inan african savanna. Scienti c data, 2(1):1{14, 2015.
Szegedy,C., Vanhoucke,V., Io e,S., Shlens,J., and Wojna,Z. Rethinking the inception architecture for computer vision. In The IEEE Conference on Computer Vision and Pattern
Recognition (CVPR), June 2016.
Taigman,Y., Yang,M., Ranzato,M., and Wolf,L. DeepFace: Closing the Gap to HumanLevel Performance in Face Veri cation. In Taigman, Y., Yang, M., Ranzato, M. & Wolf,
L. Deepface: closing the gap to human-level performance in face veri cation. In Proc.
Conference on Computer Vision and Pattern Recognition 1701{1708, pages 1701{1708,
2014.
Takeki,A., Trinh,T. T., Yoshihashi,R., Kawakami,R., Iida,M., and Naemura,T. Combining
deep features for object detection at various scales:nding small birds in landscape

images. IPSJ transactions on computer vision and applications, 8(1):1{7, 2016.
To lski,A. Drawwing, a program for numerical description of insect wings. J. Insect Sci.,
4:17, May 2004.
To lski,A. Automatic measurement of honeybee wings. In Automated Taxon Identi cationin Systematics: Theory, Approaches and Applications, pages 277{288. CRC Press, 2007.
63

Tompson,J., Goroshin,R., Jain,A., LeCun,Y., and Bregler,C. Ecient Object Localization Using Convolutional Networks. In The IEEE Conference on Computer Vision and
Pattern Recognition (CVPR), pages 648{656, 2015.
Van Horn,G., Mac Aodha,O., Song,Y., Cui,Y., Sun,C., Shepard,A., Adam,H., Perona,P.,
and Belongie,S. The inaturalist species classi cation and detection dataset. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR),
June 2018.
Vapnik,V. N. The nature of statistical learning theory. Statistics for Engineering andInformation Science. Springer-Verlag, New York, 2000.
Wah,C., Branson,S., Welinder,P., Perona,P., and Belongie,S. The caltech-ucsd birds-2002011 dataset. 2011.
Walter,D. E. and Winterton,S. Keys and the crisis in taxonomy: extinction or reinvention?
Annu. Rev. Entomol., 52:193{208, 2007.
Watson,A. T., O'Neill,M. A., and Kitching,I. J. Automated identi cation of live moths(macrolepidoptera) using digital automated identi cation system (daisy). System. Biodivers., 1(3):287{300, 2003.
Weeks,P. J. D., Gauld,I. D., Gaston,K. J., and O'Neill,M. A. Automating the identi cationof insects: a new solution to an old problem. Bull. Entomol. Res., 87(02):203{211, 1997.
Weeks,P. J. D., O'Neill,M. A., Gaston,K. J., and Gauld,I. D. Species{identi cation of
wasps using principal component associative memories. Image Vis. Comput., 17(12):861{866, October 1999a.
Weeks,P. J. D., O'Neill,M. A., Gaston,K. J., and Gauld,I. D. Automating insect identi cation: exploring the limitations of a prototype system. J. Appl. Entomol., 123(1):1{8,January 1999b.

64

Wei,X.-S., Luo,J.-H., Wu,J., and Zhou,Z.-H. Selective convolutional descriptor aggregation
for Fine-Grained image retrieval. IEEE Trans. Image Process., 26(6):2868{2881, June2017.
Werbos,P. J. Applications of advances in nonlinear sensitivity analysis. In System modelingand optimization, pages 762{770. Springer, 1982.
Wilf,P., Zhang,S., Chikkerur,S., Little,S. A., Wing,S. L., and Serre,T. Computer visioncracks the leaf code. Proc. Natl. Acad. Sci. U. S. A., 113(12):3305{3310, March 2016.
Willi,M., Pitman,R. T., Cardoso,A. W., Locke,C., Swanson,A., Boyer,A., Veldthuis,M.,
and Fortson,L. Identifying animal species in camera trap images using deep learningand citizen science. Methods in Ecology and Evolution, 10(1):80{91, 2019.
Wu,X., Zhan,C., Lai,Y., Cheng,M., and Yang,J. Ip102: A large-scale benchmark dataset
for insect pest recognition. In 2019 IEEE/CVF Conference on Computer Vision and
Pattern Recognition (CVPR), pages 8779{8788, 2019. doi: 10.1109/CVPR.2019.00899.
Xie,S., Girshick,R., Dollar,P., Tu,Z., and He,K. Aggregated residual transformations fordeep neural networks. In Proceedings of the IEEE conference on computer vision andpattern recognition, pages 1492{1500, 2017.
Yosinski,J., Clune,J., Bengio,Y., and Lipson,H. How transferable are features in deepneural networks? In Advances in neural information processing systems, pages 3320{
3328, 2014.
Zeiler,M. D. and Fergus,R.

Visualizing and Understanding Convolutional Networks.

Computer Vision { ECCV 2014, 8689, 2014. doi: 10.1007/978-3-319-10590-1. URL
http://link.springer.com/10.1007/978-3-319-10590-1.
Zhang,H., Cisse,M., Dauphin,Y. N., and Lopez-Paz,D. mixup: Beyond empirical risk
minimization. arXiv preprint arXiv:1710.09412, 2017.

65

Zhang,J. and Zong,C. Deep Neural Networks in Machine Translation: An Overview. IEEE
Intelligent Systems, 30(5):16{25, sep 2015. doi: 10.1109/MIS.2015.69. URL http://ieeexplore.ieee.org/document/7243232/.
Zhang,Z.-Q. Animal biodiversity: An outline of higher-level classi cation and survey oftaxonomic richness, volume 3148. Magnolia Press, 2011.
Zheng,L., Zhao,Y., Wang,S., Wang,J., and Tian,Q. Good practice in CNN feature transfer.
April 2016.
Zhou,B., Khosla,A., Lapedriza,A., Oliva,A., and Torralba,A. Learning deep features fordiscriminative localization. In Proceedings of the IEEE conference on computer visionand pattern recognition, pages 2921{2929, 2016.

66