Itaki et al. Progress in Earth and Planetary Science
https://doi.org/10.1186/s40645-020-00332-4(2020) 7:19

Progress in Earth andPlanetary Science

METHODOLOGYOpen Access

Automated collection of single species of
microfossils using a deep learning–
micromanipulator system
Takuya Itaki1* , Yosuke Taira2, Naoki Kuwamori2, Toshinori Maebayashi3, Satoshi Takeshima4 and Kenji Toya4

Abstract
For geochemical analysis such as stable isotope ratio, radiocarbon dating and minor element analysis for a single
species of microfossils, a large number of specimens, is required. Collecting specimens one by one under amicroscope requires enormous time and effort. In this study, we developed a device that automates these efforts
and can be used without expert knowledge. Microfossils can be accurately classified and identified to taxonomic
species level using deep learning, which is one of the learning methods of artificial intelligence (AI), and picked up
using a micromanipulator installed in the microscope with an automated motorized X-Y stage. A prototype of theclassification model AI-PIC_20181024 showed the ability to classify microfossil species Cycladophora davisiana andActinomma boreale (radiolarians) with accuracy exceeding 90% at a confidence level > 0.90. Using this method, it ispossible to collect a large number of particles with speed and accuracy that cannot be achieved by a humantechnician. Although this technology can only be used for specific species of microfossils, it greatly reduces thehand work of picking and also enables chemical analysis, such as isotope ratio and minor element analysis, for small
microfossil species for which it had been difficult to collect enough specimens. In addition to microfossils, thistechnology can be applied to other particles, with applications expected in various fields, such as medical, food,horticulture, and materials.
Keywords: Micropaleontology, Radiolarians, AI, Classification, Manipulation

Introduction
Stable isotope ratio, radiocarbon dating, and minor
elemental analyses of a single species of foraminifera,
which is a group of microfossils, require a large numberof specimens. Current sampling practice is manually collecting specimens one by one under a microscope by anexpert technician. Recent improvements in analyticaltechnologies have made it possible to carry out various
analyses on extremely small samples, but it is still difficult to secure sufficient quantities of sample for analysis.
For example, Ijiri et al. (2014) developed an inductive
high-temperature carbon reduction method that allows* Correspondence: t-itaki@aist.go.jp
1
Geological Survey of Japan/AIST, Institute of Geology and Geoinformation,
1-1-1 Hihashi, Tsukuba, Ibaraki 305-8567, Japan
Full list of author information is available at the end of the article

measuring the oxygen isotope ratio of sub-milligram
quantities of biogenic opal. As a result, it has become
possible to analyze a single species of radiolarian Spongotrochus glacialis with a relatively large body. However, it
is still necessary to collect more than 1,000 specimens
for the majority of species of radiolarians (marine zooplankton with opal skeleton) that have body size smaller
than 100 μm (< 0.1 μg/individual). In practice, it is difficult to collect enough individuals of the small radiolarianspecies for analysis, and there is expected to be interestin this never before reported data. However, before suchresearch can be undertaken, it is necessary to develop amethod for accurately classifying microfossils of arbitrary species and to collect them in large quantities.
In practical terms, one solution is to use a micro particle accumulator with a micromanipulator along with© The Author(s). 2020 Open Access This article is licensed under a Creative Commons Attribution 4.0 International License,
which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give
appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate ifchanges were made. The images or other third party material in this article are included in the article's Creative Commonslicence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commonslicence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain
permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.

Itaki et al. Progress in Earth and Planetary Science(2020) 7:19

the latest artificial intelligence (AI) technology withutilization of big data and a high-performance computer.
The micro particle accumulator can distinguish mineralparticles by conventional machine learning (Isozaki
et al., 2018), but it has not yet been possible to accurately classify microfossils of delicate and complicatedforms. On the other hand, AI using deep learning
methods has been used for distinguishing objects
appearing in images. Unlike traditional machine learningwhere a person extracts features, features are automatically extracted by the computer, making it is suitable forclassification of complicated forms like microfossils. Recently, deep learning has been used to classify volcanic
ash particles (Shoji et al., 2018) and foraminiferal tests(Mitra et al., 2019). However, in these previous methods,
a large number of microscopic images first needed to becollected.
Therefore, we have developed a system that enables
accumulating a massive number of digital images of microfossils identified to the species level by integration ofexisting technologies such as AI, digital image processing, and precise micromanipulation (AIST Press Release,2018). In this paper, we present an overview of this system and the results of a practical test with radiolarianfossils.

System overview
The system was basically designed to allow a series ofoperations from classification to collection of microfossils using three units (Fig. 1): (1) Image Collection Unit,

Page 2 of 7

(2) Classification Unit, and (3) Particle Collection Unit.
These units are connected to each other with a newly
developed communication program. Each unit of thissystem is described below.

Image Collection Unit

This unit automatically acquires microscopic images of alarge number of particles scattered on an observation
area using an electric X-Y stage microscope with computer control. In this study, “Collection Pro” from Micro
Support Co., Ltd., which is documented in Isozaki et al.
(2018), was used as a base unit. This unit allows forscanning of particles scattered on the sample tray using
a microscope with a high-resolution CCD (charge
coupled device) camera in an arbitrary area, and acquires
individual images clipped in arbitrary sizes and resolutions by image processing (Supplementary Movie 1).
Particle sizes and their coordinates on the sample tray
are recorded by the computer.
When a single particle is taken alone in the clipped
image, it is recognized as one individual. However, if aplurality of particles is overlapping, they are erroneously
recognized as one individual by image processing. To
minimize overlapping particles, a total of 60,000 dimples
with diameters of 90 and 140 μm were drilled on the tray
surface (Supplementary Figure 1), and a structure wasconstructed such that one individual particle fits in eachindentation. As a result, particles can be efficiently dispersed to some extent and overlapping can be
minimized.

Fig. 1 Schematic of the automated microfossil collection system, which is composed of three units: Image Collection Unit, Classification Unit, andParticle Collection Unit. A photo of the system is shown in the upper right. Image scanning over the tray and picking up microfossils can be seenin Supplementary Movies 1 and 2

Itaki et al. Progress in Earth and Planetary Science(2020) 7:19

Classification Unit

The Classification Unit consists of a computer operating
deep learning software and a program that facilitates
data exchange with the Image Collection Unit. This unit
has two roles: building a classification model based on alarge number of images obtained from the Image Collection Unit, and using the model to identify particles extracted by the Image Collection Unit during practicalclassification work.
Deep learning was used to classify microfossil species
from images. One of the machine learning methods isbased on a multilayered neural network, which is a computational model inspired by biological neural networks,that can automatically learn features from images. The
deep learning software “RAPID machine learning” (NECCorp.), which incorporates a convolution neural network(CNN) and can be tuned to quickly construct a classification model from a large amount of data without usinga graphics processing unit (GPU), was adopted in thissystem.
Particle Collection Unit

The target particles determined by the classification results are automatically picked up from the recorded coordinates on the sample tray by the micromanipulator ofthis unit. A vacuum suction type micromanipulator installed on the Image Collection Unit grips a particle at the
tip of a glass tube with a diameter of 30–50 μm (Fig. 2b).

Page 3 of 7

Then, the particle is carried to the collection location andreleased by stopping the vacuum.
A static-free field around the picking unit is generated
by a static electricity remover in order to avoid adsorptionof particles so that particles can be deposited. However,when a captured specimen is placed on the sample tray by
the micromanipulator, it sometimes does not separate
from the nozzle tip due to static electricity. Therefore, anadhesive sheet was placed in the collection area to help
separate particles from the nozzle. The microfossils can beremoved from the sheet with some ethanol.

Methods/Experimental
In this experiment, siliceous fossil-rich PleistoceneHolocene sediments collected from the Southern Ocean
and the Japan Sea were used. The samples were reactedwith 10% HCl and 10% H2O2 solutions in order to exclude carbonate and organic matter, respectively, andthen were sieved using 63 μm stainless mesh. Clastic
mineral particles were extracted from residues on the
63 μm mesh by the method of Itaki (2006) in order toconcentrate siliceous microfossils as much as possible.
Dried particles ranging from 63 to 250 μm were mainly
radiolarians and clastic minerals and were distributed
homogeneously on the aluminum sample tray. Although
there were many radiolarian fossils in the particles larger
than 250 μm, the target species for classification and

Fig. 2 Photographs showing microfossil pick-up and collection. a Microfossils are picked up using the micromanipulator from among various
particles scattered on a specially designed sample tray. b An extracted microfossil (Cycladophora davisiana) gripped at the tip of the vacuum-type
micromanipulator. c The assemblage of C. davisiana collected on an adhesive sheet. Scale equals 200 μm

Itaki et al. Progress in Earth and Planetary Science(2020) 7:19

collection in this study were mainly in the 63 to 250 μm
size fraction.
Digital images of particles on the sample tray weretaken with 280 × 280 dpi resolution under the epiillumination mode of the Image Collection Unit. A large
number of these images were used as the training datafor construction of the classification model using deeplearning by the Classification Unit. Generally, the training data should be composed of more than 1,000 images
in each category, including target species for picking andother kinds of particles. Usually, collecting such a training dataset by human technicians requires a lot of laborand time; however, this system can automate this step. If
a sufficient number of images cannot be collected for acategory of rare species, the training data are built out
with rotations and flips of a single image (Fig. 3).

Page 4 of 7

The classification model based on the training dataset
is created using deep learning software in the Classification Unit. Learning repetition is basically 30 epochs(Supplementary Figure 2). The accuracy of the createdmodel is tested using images that were not used for themodel construction. Although the initial accuracy of the
newly constructed model is usually low, through repetition and revision of the training dataset, higher accuracy
can be achieved.

Results and discussion
Operation processes of the system including model construction, classification, picking, and collection werechecked with radiolarian fossils in deep-sea sediment.
Radiolarian fossils with complicated and delicate siliceous skeleton are suitable as a test material for thisFig. 3 Screen shots of subsets of training data for a Actinmma boreale and b Cycladophora davisiana used in model AI-PIC_20181024

(2020) 7:19

Itaki et al. Progress in Earth and Planetary Sciencesystem owing to the difficulties of species identificationand picking their small skeletons. Test results from theprocess are noted as follows.
Training data and model construction

In classification by deep learning, all collected images
were applied to one of the previously learned categories.
The confidence level for each category of the image wasestimated, and the category with the highest value wasshown as the classification result. In this system, since
all the images acquired by the Image Collection Unit
were classified in one of several categories, it is necessary
to efficiently construct a classification model in order todistinguish the target species from various particles.
The sample particles used in this experiment consisted
mainly of radiolarians and clastic particles. Although thesample included multiple radiolarian species, we decided
to build a learning model with Cycladophora davisiana
and Actinmma boreale as the target species (Fig. 3).
In the initial constructed model AI-PIC_20181024, approximately 20,000 images of particles were selectedfrom more than 30,000 images collected by the system
and were categorized into the following 8 categories: C.
davisiana [1884 images], Cycladophora bicornis [1088images], A. boreale [1796 images], Antarctissa spp. [2129
images], Larcopyle pylomaticus [1134 images], other radiolarians [7919 images], clastic particles [1745 images],shell fragments [564 images], and reflections of illumination [2514 images]. Although C. bicornis was not a target species, it was separately categorized for
differentiation from C. davisiana due to having similarmorphology. Antarctissa spp. and L. pylomaticus werealso categorized separately in this model; however, results for these categories were not evaluated in this study
due to difficulty in discrimination because of unreliable
images from the training data.
Practical tests based on model AI-PIC_20181024 were
performed using three samples from the Japan Sea(IODP site U1422C 1H-1, 68–70 cm) and the Del CanõRise in the Southern Ocean (45.7° S, 44.4° E, 2445 m inwater depth, Piston core site DCR1PC, #39). Confidence
level (0.00–1.00) meaning certainty of correct

Page 5 of 7classification, total number of images used in classification through the scan, accuracy (number of correct classifications/number of categorized images) for C.
davisiana and A. boreale, and number of uncategorized
images with percentages are shown in Table 1.
In this practical test, the confidence level was set to0.60 for U1422, and 0.80 and 0.90 for DCR1PC. Whenthe confidence level was 0.60, uncategorized images
accounted for 11% of 2,472 images, while when it was
0.90, about 60% of all particle images were considered as
uncategorized images and were excluded from classification. The number of uncategorized images tends to behigher with increasing confidence level.
The accuracy for C. davisiana for the same DCR1PC
sample was 70% and 92% at confidence levels of 0.80
and 0.90, respectively. Thus, accuracy increased withconfidence level but dropped with the number of images
categorized for this species. On the other hand, the accuracy for both C. davisiana and A. boreale from sample
U1422C was 94% at a confidence level 0.60. Such highaccuracy for both of species despite the relatively low
confidence level of 0.60 was possibly related to the reduction in misclassification due to the low diversity ofradiolarian assemblages in this sample.
As demonstrated above, setting the confidence levelhigh results in an increased classification accuracy, while
many images are excluded due to being below the confidence level. An ideal model is one that realizes both thehigh accuracy and reduces uncategorized images, as isshown for sample U1422C with confidence level 0.60.
Although model AI-PIC_20181024 was constructedbased on training data from both the Japan Sea and theSouthern Ocean, higher accuracy may be achieved bybuilding models independently for each region or timeframe using local samples for training data.
In addition to radiolarians, classification models forother microfossil groups such as foraminifers (marine
zooplankton with CaCO3 skeleton) and diatoms (phytoplankton with opal skeletons) have been attempted using
this system. Although radiolarian species can usually beclassified by a single view from one direction, foraminifers should be observed from various directions in order

Table 1 Results of practical test for model (AI-PIC20191024) using samples U1422C, 1H-1, 68–70 cm (Japan Sea) and DCR1PC, #39(Southern Ocean)
Item

U1422C; 1H-1, 68–70 cm

DCR-1PC, #39(a) Confidence level

0.6

0.8

0.9

(b) Total number of images

2,472

11,4169,815(c) Actinomma boreale

45/48 (94%)

59/59 (100%)

41/41 (100%)(d) Cycladophora davisiana

165/176 (94%)

103/149 (70%)

54/59 (92%)(e) Uncategorized image

267 (11%)

4,335 (38%)

5,768 (59%)(a) confidence level (0.00–1.00) meaning the certainty of correct classification, (b) total number of images used in classification through the scan, (c–d) accuracies(number of correct classification/number of categorized images) for Cycladophora davisiana and Actinmma boreale, and (e) number of uncategorized images withpercent of total

Itaki et al. Progress in Earth and Planetary Science(2020) 7:19

to recognize the 3-dimensional characteristics of theskeleton. This means that a mechanism for distinguishing foraminiferal species from multi-directional images
is required. In the case of diatoms, digital images using ahigher-power microscopic lens using composite focusing
image programs are needed to capture their smaller
skeletons. To resolve these issues for various microfossil
groups, further development of this system is expected
as a next step.
Particle collection

Using the model AI-PIC_20181024, C. davisiana was selected by the Classification Unit from the residue of thetest sample spread on the sample tray by the Image Collection Unit. The classified particle images were displayed on the operation monitor of the Image Collection
Unit for checking, and the misclassified particles were
excluded from the target by a human technician in order
to avoid including wrongly identified particles. Correctlyclassified images can be organized and stored as additional training data, which can later be used to construct a more accurate classification model.
After the classification was completed, all identified C.
davisiana specimens were sequentially picked up fromtheir recorded coordinates on the tray by the micromanipulator (Supplementary Movie 2). In advance of collecting the particles, it is necessary to accurately set theposition information of the suction nozzle tip of the micromanipulator, the approach direction to the particles,
and the collection area of the picked up particles. The
accuracy of this setting is important for reducing pickup failure. In this experiment, 70–80% of the identified
particles could be collected successfully. Failure seemed
to occur when there was a small distance between the
tip of the nozzle and the target. Improving collection efficiency is an improvement point for the system.
In this practical test, using samples that contained sufficient numbers of the target species, the picking speed
was about 120 specimens per hour. If the sample contains sufficient numbers of the target species, the target
number of specimens for analysis may be reached with asingle sample, but it may be necessary to repeat theprocess with additional samples if the number collected
is not sufficient.

Conclusions
An automated microfossil pick-up system with implementation of AI technology was newly developed, andresults of a practical test of this system confirm the practical use of a classification model with sufficiently highaccuracy. Using this automated system, microfossils
could be collected on species level, whereas previously, ahuge amount of time and labor was required to collectthe samples by hand. The model AI-PIC_20181024 used

Page 6 of 7in this experiment can be used to classify two radiolarianspecies, C. davisiana and A. boreale, with accuracies ofmore than 90% at a confidence level of 0.90, but almost
half of the images remain uncategorized. The number of
uncategorized images decreased at lower confidence
levels (0.60 and 0.80); however, the accuracy also tended
to deteriorate at these confidence levels. In order to collect target microfossils more efficiently, it is important
to construct an excellent classification model that has
high accuracy even at low confidence levels. Because thesystem and classification model reported in this paper
are still in the prototype stage of development, both thedevice and model are improved for the practical uses.
In addition to applications to microfossils, this system
can be applied to the classification and extraction ofmineral particles. This system can be adapted to various
specialized sorting needs, such as preparation of highpurity materials in the steel industry, sorting of plantseeds and fish eggs, removal of impurities and contaminants in foods, and sorting of abnormal cells, embryos,
or platelets. In order to achieve workable systems inthese fields, further development is needed to link AI
and microscope systems and to develop a separator suitable for each type of particle.

Supplementary information
Supplementary information accompanies this paper at https://doi.org/10.
1186/s40645-020-00332-4.
Additional file 1. : Supplementary Movie 1: Collection of particle images
through scanning on the sample tray. Particles detected by image
processing are shown as red areas, and they are clipped as individual
images. (PPTX 27074 kb)
Additional file 2. : Supplementary Movie 2: Picking up microfossils fromthe sample with the micromanipulator and depositing them on thecollection area. Three trials are recorded. The first two collections weresuccessful. The third one failed, likely because the tip of the nozzle was
not brought close enough to the target. (PPTX 21185 kb)
Additional file 3. : Supplementary Figure 1: Dimensions of the particle
dispersion tray. A total of 60,000 dimples with diameters of 0.14 mm and0.09 mm were drilled on the tray surface. (a) Upper view of the plate, (b)cross-sectional view of the plate and (c) cross-sectional view of dimples
with diameter of 0.14 mm and 0.09 mm. All dimensions are in millimeters. (AI 355 kb)
Additional file 4. : Supplementary Figure 2: Diagram showing error
values for each learning repetition through 30 epochs on the deeplearning software RAPID.
Acknowledgements
We express sincere gratitude to Minoru Ikehara of Kochi University and Akira
Ijiri and Tasuhiko Hoshino of JAMSTEC for their helpful suggestions andadvice. We are also grateful to Ryota Kakishita and Hitomi Yamazaki for theirassistance in the laboratory experiments. This manuscript was greatly
improved by comments from the reviewers.
Authors’ contributions
TI conceived and conducted the experiments. YT and NK coordinated the AI
learning program. TM developed a microscope and micromanipulation
system, and ST and KT developed the system program including image
processing. All authors reviewed the manuscript. The authors read andapproved the final manuscript.

Itaki et al. Progress in Earth and Planetary Science(2020) 7:19

Funding
This work was supported by JSPS KAKENHI Grant Numbers 17H06318 and
18H01329. Patent applications have been awarded for the MICAD system
reported in this study since August 31, 2018 for Japan [Patent application
2018-163981] and since May 10, 2019 for international application [PCT/
JP2019/018730]. Patent applicants: NEC corporation and Advanced Instituteof Science and Technology (AIST). Name of inventors: Naoki Kuwamori,
Yosuke Taira, Takuya Itaki, Toshinori Maebayashi, Satoshi Takeshima and Kenji
Toya.
Availability of data and materials
The training datasets used in the current study are available upon
reasonable request from the corresponding author.
Competing interests
The authors declare that they have no competing interests.
Author details
1
Geological Survey of Japan/AIST, Institute of Geology and Geoinformation,
1-1-1 Hihashi, Tsukuba, Ibaraki 305-8567, Japan. 21st Government and Public
Solutions Division, NEC Corporation, Tokyo 108-8001, Japan. 3Head Office
Sales Department, Micro Support Co., Ltd., Shizuoka-City, Shizuoka 422-8036,
Japan. 4Visual System Division, Mitani Corporation, Tokyo 171-0033, Japan.
Received: 20 August 2019 Accepted: 27 April 2020

References
AIST Press Release (2018) New technology utilizing artificial intelligence has been
established for accurate identification and sampling of microfossils - enablinghigh-speed automatic analysis of geological strata, https://www.gsj.jp/en/
research/topics/pr20181203.html (Summary of the AIST Press Release on
December 3, 2018). Accessed 17 Aug 2019
Ijiri A, Yamane M, Ikehara M, Yokoyama Y, Okazaki Y (2014) Online oxygenisotope analysis of sub-milligram quantities of biogenic opal using theinductive high temperature carbon reduction method coupled with
continuous-flow isotope ratio mass spectrometry. J Quaternary Sci 29:455–
462. https://doi.org/10.1002/jqs.2716
Isozaki Y, Yamamoto S, Sakata S, Obayashi H, Hirata T, Obori K, Maebayashi T,
Takeshima S, Ebisuzaki T, Maruyama S (2018) High-reliability zircon separation
for hunting the oldest material on Earth: an automatic zircon separator withimage-processing/microtweezers-manipulating system and double-stepdating. Geoscience Frontiers 9:1073–1083
Itaki T (2006) Elutriation technique for extracting radiolarian skeletons from sandy
sediments and its usefulness for faunal analysis. Radiolaria 24:14–18
Mitra R, Marchitto TM, Ge Q, Zhong B, Kanakiya B, Cook MS, Fehrenbacher JS,
Ortiz JD, Tripati A, Lobaton E (2019) Automated species-level identification ofplanktic foraminifera using convolutional neural networks, with comparison
to human performance. Marine Micropaleontol 147: 16-24, doi.org/https://doi.org/10.1016/j.marmicro.2019.01.005.
Shoji D, Noguchi R, Otsuki S, Hino H (2018) Classification of volcanic ash particles
using a convolutional neural network and probability. Scientific Reports 8:
8111. https://doi.org/10.1038/s41598-018-26200-2

Publisher’s NoteSpringer Nature remains neutral with regard to jurisdictional claims inpublished maps and institutional affiliations.

Page 7 of 7