sensors
ArticleEstimating 3D Leaf and Stem Shape of Nursery
Paprika Plants by a Novel Multi-Camera
Photography System
Yu Zhang, Poching Teng, Yo Shimizu, Fumiki Hosoi and Kenji Omasa *Graduate School of Agricultural and Life Sciences, The University of Tokyo, 1-1-1, Yayoi, Bunkyo-ku,
Tokyo 113-8657, Japan; 0491342004@mail.ecc.u-tokyo.ac.jp (Y.Z.); 8015764991@mail.ecc.u-tokyo.ac.jp (P.T.);
ayosh@mail.ecc.u-tokyo.ac.jp (Y.S.); ahosoi@mail.ecc.u-tokyo.ac.jp (F.H.)* Correspondence: aomasa@mail.ecc.u-tokyo.ac.jp; Tel.: +81-3-5841-5340; Fax: +81-3-5841-8175
Academic Editors: Changzhi Li, Roberto Gómez-García and José-María Muñoz-Ferreras
Received: 12 April 2016; Accepted: 6 June 2016; Published: 14 June 2016

Abstract: For plant breeding and growth monitoring, accurate measurements of plant structure
parameters are very crucial. We have, therefore, developed a high efficiency Multi-Camera
Photography (MCP) system combining Multi-View Stereovision (MVS) with the Structure from
Motion (SfM) algorithm. In this paper, we measured six variables of nursery paprika plants andinvestigated the accuracy of 3D models reconstructed from photos taken by four lens types at four
different positions. The results demonstrated that error between the estimated and measured values
was small, and the root-mean-square errors (RMSE) for leaf width/length and stem height/diameter
were 1.65 mm (R2 = 0.98) and 0.57 mm (R2 = 0.99), respectively. The accuracies of the 3D modelreconstruction of leaf and stem by a 28-mm lens at the first and third camera positions were thehighest, and the number of reconstructed fine-scale 3D model shape surfaces of leaf and stem is themost. The results confirmed the practicability of our new method for the reconstruction of fine-scale
plant model and accurate estimation of the plant parameters. They also displayed that our system isa good system for capturing high-resolution 3D images of nursery plants with high efficiency.
Keywords: 3D modeling; leaf; stem; accuracy; MCP system; stereovision; MVS; SfM

1. Introduction
For plant breeding and growth monitoring, accurate measurements of plant structure parametersand plant functioning are very crucial [1,2]. Over the past three decades, 2D imaging has been applied
to a variety of aspects, ranging from structural analysis [3], stomata movements and photosynthetic
diagnosis [4–6], herbicide control [7,8], disease detection [9] and yield estimation [10]. However, as 2D
imaging is not robust against the occlusion of plant organs, such as overlapping leaves and stems andchanging shapes during the measurement, most of these applications are either for large-scale analysis
or for simple plant variables’ measurements at the early growth stage [11]. Recently, 3D imaging
technologies for the robust measurement of plant breeding and growth parameters have emerged andbeen applied for solving this problem owing to the advance of sensing technology [2,12,13].
Among the applications of 3D imaging technology in agriculture, 3D model construction forplants [2,14] and properties’ estimation for plant growth [15,16] are the most noticeable. Today,
there are many commonly-known data acquisition methods for these applications, including active
and passive methods. Among the active sensing technologies, scanning LiDARs (Light Detection andRanging) are usually used for constructing accurate and detailed 3D models, but such systems areusually expensive and complex [14,17]. In our previous research, we have evaluated the performanceof portable scanning LiDAR for estimating the leaf inclination angle distribution, leaf area density

Sensors 2016, 16, 874; doi:10.3390/s16060874

www.mdpi.com/journal/sensors

Sensors 2016, 16, 874

2 of 18

profiles, vertical plant area density, forest structure, water storage, etc. [2,16,17]. However, scanning
LiDAR has the shortcomings of not only high cost and much time consumed, but also different sensor
properties and robustness against background illumination [18]. Moreover, as leaf shapes are difficult
to capture accurately if the leaves move because of air movement, the scanning LiDAR measurements
must be performed without any disturbance of air movement [19]. Apart from the above active
technologies, depth-sensing cameras [20,21] are more easily deployable and a less complicated modeof 3D data acquisition, as they enable acquiring image data in real time about texture or lighting
conditions. However, although the cameras are low cost and easily applied, the captured data are
difficult to directly apply because of their low resolution and high noise.
Among the passive sensing technologies, Shape-from-X and stereovision are the most widespreadmethods of acquiring 3D data. They have been successfully applied in indoor and outdoor studies.
For example, Omasa et al. [22] showed the 3D microscopic measurement of intact petunia seedlings formeasuring shape and growth with a modified shape-from-focus method that incorporated a linear
regression operator for 3D reconstruction. As for stereovision, it is also widely applied to reconstruct
3D models of plants [15] under indoor conditions. For example, some researchers [23] showed leaf wilt
detection by stereovision. For outdoor imaging in a limited scope or at larger scales, there was alsomuch work in the literature that introduced the accuracy of stereovision [24,25] and the classificationof various plants with experimental variations of the environmental factors [26]. For example, this wasapplied for measuring outer leaf inclination angles of soybean plants and tracking their diurnal andnocturnal leaf movements [27]. Moreover, it was also used for guidance and navigation in the field [28].
In short, these applications were mostly conducted by using binocular stereovision. However, there aremany limitations. Firstly, similar to 2D imaging technology, self-occlusion is still considered one of thelimitations in stereovision in conducting various visual tasks, such as 3D scene reconstruction, target
recognition, stereo matching, visual tracking, and so on [2,29]. Self-occlusion occurs when one partof an object is occluded by another part of the object from the camera’s view, which will affect theresults of the above tasks. Besides, stereovision is easily affected by wind or strong sunlight in outdoor
conditions. Stereo matching and accuracy vary with the kind of algorithm used. The local matchingalgorithm is truly efficient, but less accurate than the global one, which could be, computationally,
very expensive [30]. Finally, its performance is poor for close-range observation of surfaces because of
the homogeneous surface texture, which produces pockets of missing depth information of the plants.
In general, according to the comparison of the above active and passive sensing technologies
under indoor or outdoor illumination conditions, we can find out that the best approaches for 3D
reconstruction are often focused on a particular application and cannot be easily adapted for conducting
complex measurements. For stereovision, it can be a promising approach if we can improve its
self-occlusion and accuracy problems. Thus, in this study, we applied a Multi-View Stereovision (MVS)
method to design a novel image acquisition system with several low-cost digital cameras under indoor
illumination for 3D reconstruction.
The MVS method originated as a natural improvement of the two-view stereo algorithms.
Instead of capturing two photographs from two different viewpoints, MVS would capture more
viewpoints in between to increase robustness, e.g., to image noise or surface texture [31–33]. As only
one viewing perspective makes modeling difficult and some areas might not have been covered
sufficiently, the images captured from different and overlapping views can effectively reconstruct a
scene. Harwin and Lucieer [34] also suggest data collection from different perspectives. However,for determining the intrinsic camera parameters and the position of the corresponding points from
uncalibrated images taken by several cameras, other techniques to reconstruct both camera positions
and 3D points are necessary. A novel solution is proposed to improve surface representation andconstruct 3D plant models through automated feature extraction and key point matching acrossimages based on the Structure from Motion (SfM) algorithm. SfM is a stereovision technique ofsimultaneously estimating camera positions and orientations and 3D scene structure from an unordered
image dataset [35]. This algorithm can mitigate the self-occlusion problem in the stereovision method.

Sensors 2016, 16, 874

3 of 18

Moreover, compared to scanning LiDAR and other methods, the cost of producing SfM point clouds isvery low. Thus, there is a great potential in applying the SfM algorithm to reconstruct the 3D model.
In fact, the SfM algorithm and related techniques have been introduced and developed in the last two
decades. For example, the Scale-Invariant Feature Transform (SIFT) operator [36] was introduced toprovide a potent illustration of characteristics in situ and allows significant characteristics in otherperspectives to be contrasted and matched. Bundle adjustment was applied to generate a clusterof 3D coordinates of matching characteristics. The SfM algorithm was originally applied to manyground-based applications. It was also applied widely in reconstructing cultural and urban features
for 3D modeling [37–39]. Recently, there were some geographic applications based on aerial platforms
for outdoor features [27,40]. However, the SfM method for plants has been demonstrated to be moredifficult than for other characteristics due to their more complicated and irregular structures [41–43].
The SfM algorithm has also some limitations in dealing with missing data. In this study, therefore,
we have studied the combination of SfM and MVS algorithms for optimizing photo consistency andauto-calibration with high accuracy.
According to our investigation, no studies have specifically reported 3D vegetation modelingand the accuracy obtained using the SfM and MVS algorithms in indoor conditions, especially fornursery plants [44–47]. Consequently, this study is necessary to develop a low cost and high accuracy
Multi-Camera Photography (MCP) system to generate a 3D model for nursery plants based on
the MVS and SfM algorithms. The novelty of our method lies in the integration of SfM and MVS
algorithms in indoor conditions for nursery plants. According to our method, we designed our
MCP-SfM image acquisition system and demonstrated its feasibility and effectiveness through many
experiments. Our method has solved some problems or limitations of other methods, such as highcost, self-occlusion, low accuracy, information missing, noises, and so on. Therefore, our proposed
approach is innovative. Our objective is to estimate and compare plant parameters with different
cameras and lenses for 3D imaging. We will introduce some metrics for the qualitative evaluation of
3D imaging and its accuracy for different lenses used for 3D shape reconstruction under four different
positions. We will finally discuss the accuracy discrimination of leaf and stems properties, as well as
the characteristics and limitations of the proposed system.
2. Materials and Methods
2.1. Plant Material
Paprika seedlings were grown in a combined black four-square nursery box(35.5 cm length ˆ 28.0 cm width ˆ 7.6 cm high) with 9 small pots in an environmentally-controlled
growth chamber for over 3 weeks after sowing. The pots were filled with artificial soil (mixture ofvermiculite and perlite, 2:1, v/v). Plants were watered every two days with a 1:1000 dilution of
HYPONeX nutrient solution (HYPONeX Japan Corp., Ltd., Osaka, Japan). They were illuminated for
12 h each day by LEDs and fluorescence lamps at a photosynthesis photon flux density (PPFD) of
50 mmol¨ m´2 ¨ s´1 . The PPFD ratio of red and blue light was 4:1. The air temperature was 26.5 ˝ C for
16 h in the day and 24 ˝ C for 8 h at night. The relative humidity was 70% in the day and 90% at night.
The intact nursery paprika plants were measured in the experiments.
2.2. Multi-Camera Photography-Structure from Motion System
According to the principle of MVS, the depth from images can be recovered by corresponding two
or more perspectives of the same object. The result of this process is a 3D point cloud, in which each 3D
point matches to a pixel in one of the photos. In previous studies [25–27], the binocular stereo method
was usually applied to two photos captured with two cameras divided by a horizontal distance known
as the “baseline”. In our system design, 4 cameras are used to compose the MCP-SfM system forobtaining the 3D point cloud data of nursery paprika plants’ structure. The distance between lensand plant is about 35 cm. The MCP-SfM system is able to obtain a high resolution by calibrating theSensors 2016, 16, 874

4 of 18

cameras and images corresponding the 3D point coordinates and locations in actual measurement
scales. At the beginning, AF (Auto Focus) mode was used. Then, MF (Manual Focus) mode was used
for capturing the clear photos after adjusting the focus. The exposure settings are ISO100-6400 before
capturingthe image; we have calibrated each camera for lens distortion compensation. Alongwith theSensors 2016, 16, 874
4 of 17conditions, the camera settings can provide optimal results.
ISO100-6400 before capturing the image; we have calibrated each camera for lens distortion
Figure 1 illustrates
schematic
view ofcombined
system.
high resolution
compensation.
Along awiththe conditions,
thethecamerasettingsMCP-SfM
can provide
optimalFour
results.
cameras (5184ˆ3456
pixels;
Canon
EOS
Kiss
X7,
Canon
Industrial
Co.,
Ltd.,
Tokyo,
Japan) with
Figure 1 illustrates a schematic view of the combined MCP-SfM system. Four high resolution
4 lensesof different
focal
lengths(24EOS
mm,
28X7,
mm,
35 mm,
50 mm)
wereTokyo,
used Japan)
for thewith
3D 4range
cameras(5184 × 3456
pixels;
Canon
Kiss
Canon
IndustrialCo., Ltd.,lenses ofdifferent focal
lengths
mm, positions
28 mm, 35(No.
mm,1 to
50 4)
mm)
were
used for bracket
the 3D range
acquisitionof individual
plants.
The(24
cameraon theadjustablewere fixedacquisitionof atindividual
plants.angles.
The camera
positions(No. 11 showsto 4) onthe
thedirections
adjustableof
bracketwereon thesame side
four different
The leftof Figurethe lens
at each
˝
fixed
on
the
same
side
at
four
different
angles.
The
leftof
Figure1shows
thedirections
of
the
lens
at
camera position. Positions 1 to 4 were 45 apart from each other in terms of vertical arrangement.
eachcameraposition.
Positions
1
to
4
were
45°
apart
from
each
other
in
terms
of
verticalarrangement.
In Position 3, the view direction of the camera was almost horizontal and can capture the entire object.
In Position 3, the view direction of the camera was almost horizontal and can capture the entire object.
The camera-to-leaf distance was between 35 and 40 cm. In order to obtain a high resolution view of
The camera-to-leaf distance was between 35 and 40 cm. In order to obtain a high resolution view ofthe leaf and stem, the distance between the camera lens and the plants was slightly different for eachthe leaf and stem, the distance between the camera lens and the plants was slightly different for eachcameraposition.
cameraposition.

Figure 1. MCP-SfM system and the overall workflow for 3D modeling and measurement.

Figure 1. MCP-SfM system and the overall workflow for 3D modeling and measurement.

Image acquisition was done under indoor lighting conditions by varying the camera shutter
Image acquisition
was done underindoor lighting
conditions
varyingtheincamera
shutter speed.
speed.
During the experiment,
fluorescencelamps illuminate
thebyentire
plant
the four-square
Duringtheexperiment,
fluorescence
lamps
illuminate
theentireplant
inthefour-square
nursery box.
nursery box. The light adjustment exit was set so that the entire plant could be uniformly illuminated.
The light
adjustment
wasofset
thatequably
the entire
uniformly
In order to
In orderto take theexit
photostheso
plant
fromplant
a 360°could
view, bea turntable
andilluminated.
timer with a stepper
˝ view,and
were used.
Duringmeasurement,
remote automatic
shutter
werealso usedmotor
take motor
the photosof theplanttheequably
from athe
360monitor
a turntable
and timer
witha stepper
the During
photographyof all cameras. theThemonitor
images are
at 5184
horizontal
3456 wereverticalpixelsperwere forused.
the measurement,
andtakenremoteautomatic
shutteralso
used
for the
frame
with
8-bit
resolution
inthesystem
by
a
Canon
EOS
Kiss
X7
cameraequippedwithphotography of all cameras. The images are taken at 5184 horizontal 3456 vertical pixels per aframe
filter (Kenko
Circular by
PL a58Canonmm forEOS
reflection
removal,Kenko
Tokina withJapanaCo.,
Ltd.,
with polarization
8-bit resolution
in the system
Kiss X7
cameraequipped
polarization
Tokyo, Japan). These image datasets will be input into PhotoScan (Agisoft LLC, St. Petersburg,
filter (Kenko Circular PL 58 mm for reflection removal, Kenko Tokina Japan Co., Ltd., Tokyo, Japan).
Russia) for calibration processing and 3D model building on the computer (see Figure 1). Besides, as
These image datasets will be input into PhotoScan (Agisoft LLC, St. Petersburg, Russia) for calibration
the data collection and processing need more time, we used an automatic shutter and turntable forprocessing
and acquisition
3D model ofbuilding
the computer(see Figure1). Besides,
as the In
data
continuous
images on
to improve
the photographyacquisition
efficiency.
thiscollection
way, we andprocessing
needtime, wecapture
used anautomatic
and turntable
continuous
acquisition ofwere able
to moreintermittently
one
photo pershutter
3 s (rotationtime) by forcontrollingthe automatic
imagesto improvetheabout
photography
In thisway, we were able to intermittently
turntable.
In total,
8 min wereacquisition
needed for efficiency.
a rotation eachtime.

capture one photo per 3 s (rotation time) by controlling the automatic turntable. In total, about 8 min
The MCP-SfM
Approach
were2.3.
neededfor a rotationeach time.
The MCP-SfM approach can be used to obtain 3D data of objects and to calibrate a cluster of

2.3. TheMCP-SfM
Approachcamerapositionsand 3D point locations for each camera track [40,48,49]. In order to understand theapproachwell, wecan conduct
approach,estimating
for aandpair toof images
at aa time,
The MCP-SfM
approachcananbeincremental
used to obtain3D dataof objects
calibrate
cluster ofrather than matching all of the images for all of the cameras and tracks at once. The first step was tocamera positions and 3D point locations for each camera track [40,48,49]. In order to understand theestimate the camera positions and 3D points for a single pair of images. The original pair should have
many feature correspondences, but also a baseline, so that the 3D point locations of the observed

Sensors 2016, 16, 874

5 of 18

approach well, we can conduct an incremental approach, estimating for a pair of images at a time,Sensors 2016, 16, 874
5 of 17rather than matching all of the images for all of the cameras and tracks at once. The first step was toviews aretheincamera
the optimumThen,
3D forpoint
coordinates
and locations
are calculated
forhavetheestimate
positionsstate.
and 3D
pointsa singlepair of images.
The originalpair should
selected
image
observed
with
the
most
tracks.
The
next
step
is
to
add
the
tracksobserved
by
the
next
many feature correspondences, but also a baseline, so that the 3D point locations of the observed views
camerasthe optimization.
A track
can beaugmented
if itlocations
is observed
at least one
cameraarein theinto
optimum
state. Then,
3D point
coordinates
and
areby
calculated
forotherthe selectedandiftriangulating
the
tracksoffersanoptimum
estimationof
theirlocations.
This
process
isimage observed with the most tracks. The next step is to add the tracks observed by the next cameras
reiterated,
one
image
at
a
time,
until
there
are
no
remaining
imagestobeperceived
by
any
of
the
into the optimization. A track can be augmented if it is observed by at least one other camera andgenerated
3D points.
To offers
optimizethe objectivefunction
at eachrepetition,
we use isthebundleif
triangulating
the tracks
an optimum
estimationof theirlocations.
This process
reiterated,adjustment
algorithm
[50],
which
was
employed
by
using
an
SfM
platform,
namelyMicrosoft
one image at a time, until there are no remaining images to be perceived by any of the generated
PhotoSynth.
3D
points. To optimize the objective function at each repetition, we use the bundle adjustment
The resulting
SfM
of by
an using
accordant
arbitrarycoordinate
system, PhotoSynth.
which should be
algorithm[50], whichwasconsists
employedan SfM
platform,
namely Microsoft
georeferenced
andconvertedtorealcoordinates.
Inlabconditions,
SfM
datasetsconsist shouldof a set beof
The resulting SfM consists of an accordant arbitrary coordinate system, which
normal referenced
pointwith additional
informationfrom thephotographs,
georeferenced
and coordinate
converted torealdata
coordinates.
In lab color
conditions,SfM datasets
consistof a setsuchasa
Rubik’scube(see
Figure2).
The
keypointsgenerated
from
the
SfM
outputcanbelocatedof normal referenced coordinate point data with additional color information from the photographs,
with
X,
Y
and
Z
coordinates.
Takinganexample
of
using
a
pair
of
imagesbythe
SfM
approach,thesuch as a Rubik’s cube (see Figure 2). The key points generated from the SfM output can be locatedgeneralsolving theTaking
SfM problem
is firstly
to estimate
structure
up to awithX, process
Y and Zfor
coordinates.
an example
of usinga pair theof images
byand
the motion
SfM approach,perspective
transformation
using
the
algebraic
methodorthefactorization
methodproposedby
the general process for solving the SfM problem is firstly to estimate the structure and motion up

Mi
Tomasiand Kanade[51]. For example,
wealgebraic
need to estimate
thethem factorization
matrices
2 ×4 projection
toa perspective
transformation
using themethod or
methodproposedby
TomasiandandtheKanadeFor example,
we needfrom
to estimate
m 2 ˆ 4 projection matrices M
S j (structure)(motion)n 3D [51].
positionsthe m ×then 2D correspondences sij (onlyi(motion) and the n 3D positions S j (structure) from the m ˆ n 2D correspondences sij (only allowing forallowing for translation and rotation between the cameras). This gives 2 m ×n equations withtranslation and rotation between the cameras). This gives 2 m ˆ n equations with
8m ` 3n unknowns
unknowns that can be solved using the algebraic method or the factorization method.
8m +can3n bethatsolved using the algebraic method or the factorization method.
ssij “M
1,…,n
Mi SS j ,, ii =“1,…,m;
1, ..., m;j =j “
1, ..., n
ij

i j

(1)
(1)

Figure 2. Methods for SfM for 3D model reconstruction.
Figure 2. Methods for SfM for 3D model reconstruction.

Some measurements are computed with extractive points from the SfM datasets that areSome measurements are computed with extractive points from the SfM datasets that are matchedmatched to parameter estimations (number of points, point heights, etc.). Then, we can convert fromto parameter estimations (number of points, point heights, etc.). Then, we can convert from perspectives
perspectives to metrics by auto-calibration and minimize reprojection errors by optimizing thelocation of 3D points and the camera by applying bundle adjustment.
Similarly, as for the MCP-SfM approach with multiple images (see Figure 2), we need to find acommon scale so that multiple images can be combined. One method is to use the Perspective NPoint (PnP) algorithm [52], where we try to solve for the position estimation of a new camera usingSensors 2016, 16, 874

6 of 18

to metrics by auto-calibration and minimize reprojection errors by optimizing the location of 3D points
and the camera by applying bundle adjustment.
Sensors
2016, 16, as
874for the MCP-SfM approach with multiple images (see Figure 2), we need to
6 of
17 a
Similarly,
findthe scene
pointswe havealreadyfound.
SfM methods
can robustly
characteristics
commonscale
so thatmultiple
imagescanTypicalbe combined.
One method
is to corresponduse the Perspective
N-Point
in asalgorithm
many pairsof inputas position
to recover
the estimationsof therelatively
(PnP)
[52],
wherephotos
we tryastopossible,
solve forsotheestimationof a newcamerausingaccurate
the scene
positions
betweenphotopairs.
Then,SfM
the methods
bundle adjustment[53]
calculates characteristics
a maximum likelihood
pointswe havealreadyfound.
Typicalcan robustly
correspondin as many
estimation
the camera
positionspoint
positions,after initialization
by a subsetof the coupled
pairs
of inputofphotos
as possible,
soand
as to
recoverthe estimationsof the relatively
accurate
positionsestimations.
between photo pairs. Then, the bundle adjustment [53] calculates a maximum likelihood estimation of
In thismethod,and
insteadof positions,increasing after
a singlecamera intoby
theoptimization,
we increased
severalthe camera
positionspoint
initializationa subsetof the coupled
estimations.
cameras
at
a
time.
As
for
selectingwhichcamerastoincrease,
we
originally
need
to
get
the
cameraIn this method, instead of increasing a single camera into the optimization, we increased several
with the most correspondence characteristics, C, to the subsistent 3D point clouds, then increase any
cameras at a time. As for selecting which cameras to increase, we originally need to get the camera with thecamera with at least 75% C correspondences to the subsistent 3D point clouds. Generally, key point
most correspondence characteristics, C, to the subsistent 3D point clouds, then increase any camera with atcorrespondence is very difficult when working with plant characteristics due to leaf sizes and gaps,
least 75% C correspondences to the subsistent 3D point clouds. Generally, key point correspondence is very
repeating and inconsonant geometric structures, etc. [53]. Accordingly, some parts with few surface
difficult when working with plant characteristics due to leaf sizes and gaps, repeating and inconsonant
features may not be well captured; similarly, other parts with complicated geometric structures may
geometric structures, etc. [53]. Accordingly, some parts with few surface features may not be well captured;
not be well mapped or appear with radial distortion. Therefore, we implement the SfM algorithm tosimilarly,
withcomplicatedgeometric
structures
may not
be well
mappedor appearwithgenerateothermoreparts
precisepoint
clouds andsurface models
at greater
detailin spatial
applications
thanradialdistortion.
Therefore,
we
implementtheSfM
algorithm
togenerate
more
precisepoint
cloudsandthe previous methods.
surface models at greater detail in spatial applications than the previous methods.
2.4. 3D Model Processing of Nursery Plants
2.4. 3D Model Processing of Nursery Plants
Figure 3 shows the original photos of paprika plants taken by the MCP-SfM system. In
Figurepaprikataken bytheatMCP-SfM
system.
In Figure
Figure3A,3 shows
one of thethe original
original photos
photos ofwastaken plants
by a 28-mm
lens
the 1st camera
position.
The3A,
onepaprikaof the canopy
originalandphotos
was
taken
by
a
28-mm
lens
at
the
1st
cameraposition.
The
paprika
canopyleaf had a very high resolution. In Figure 3B, one of the original photos was taken
andhad a very
high
Figure 3B,of thestemoriginal
photostaken
by a 28-mm
lens
byleafa 28-mmlens atthe resolution.
3rd cameraIn
position.
Theonepaprikashowsa verywas
highresolution,
as wellat the3rdreferences.
camera position. The paprika stem shows a very high resolution, as well as the references.
as theFigure 3. Original photos of paprika plants taken by the MCP-SfM system. (A) One of the original
Figure3. Original photos of paprika plants taken by the MCP-SfM system. (A) One of the original
photos taken by a 28-mm lens at the 1st camera position; (B) One of the original photos taken by a 28photos taken by a 28-mm lens at the 1st camera position; (B) One of the original photos taken by amm lens at the 3rd camera position.
28-mm lens at the 3rd camera position.

Based on the MCP-SfM approach, the uncalibrated original image data were processed andBased on the MCP-SfM approach, the uncalibrated original image data were processed andcalculated by PhotoScan. In summary, there are 5 steps of image processing during the 3D model
calculated by PhotoScan. In summary, there are 5 steps of image processing during the 3D model
rebuilding, as shown in the middle of Figure 1. As we know, the data taken from 4 different
rebuilding, as shown in the middle of Figure 1. As we know, the data taken from 4 different measuring
measuring views had individual coordinate systems. First of all, in order to make 4 cameras have theviews
coordinate
systems.
First obtainedof all, in fromorder4tomakepositions
4 cameras
havethe same 3D
samehad3D individual
coordinate system,
the imagedatasets
camerawere
co-registeredcoordinate
system, theimage datasets
obtainedfrom 4 camera
positions were co-registered into theinto the orthogonal
coordinates
and aligned
by location
identification.
orthogonal
coordinatesandalignedbylocation
identification.
Because of the estimated matching points and camera positions, the datasets are registeredBecauseof the estimatedmatching
and camera
positions,the datasets
are registeredthroughan accurate
conversion.
Then, thepoints
conversion
was repeatedly
extracted
by selecting
in turnthroughan accurate
conversion.
theandconversion
was
repeatedly
extracted
by selecting
turnmatching
points fromthe imageThen,datasetgetting thebesttranslation
and rotation
matricesin
thatmatching
pointsfromthe imagegettingthebetween
best translation
androtationthat
minimizethe error
estimation
ondatasetaccountandof thedistancethem. This
processwasmatrices
conducted
minimizetheerror
estimationon
accountofthedistance
betweenthem.
Thisprocess
wasconducted
for all couples of the image dataset. Then, the colored point clouds of the stereo image data captured
producedwithimage
4 calibrated
stereocamerasby thepoint
MSV clouds
and SfMofalgorithms
calibrating
theforwereall couplesof thedataset.
Then,
the coloredthe stereoafter
image
data captured
stereo camera system. These points are often located by X, Y and Z coordinates in the 3D coordinate
system. The MSV algorithm starts from the calibrated stereo images and a set of tracks. For properly-Sensors 2016, 16, 874

7 of 18

were produced with 4 calibrated stereo cameras by the MSV and SfM algorithms after calibrating
the stereo camera system. These points are often located by X, Y and Z coordinates in the 3D
coordinate system. The MSV algorithm starts from the calibrated stereo images and a set of tracks.
For properly-calibrated stereo cameras, the non-distortion of stereo images is actually combined withrectification. The process computed disparity images from incoming stereo images using the blockmatching algorithm. The disparity images were generated using a local correlation algorithm based on
Sum of Squared Differences (SSD). It was optimized for efficiency and has a number of validation steps
to prune disparity. Note that the extracted 3D point clouds were entangled with redundancies, a large
number of outliers and noise. Fortunately, the additional information provided by the calibrated
images can be exploited to help mesh and surface reconstruction later. After exporting 3D point
data, the dense point cloud model was reconstructed in four steps, which includes calculating thedepth information for each camera, capturing dense point cloud correspondences and reconstructing
the structure from motion and dense 3D point clouds. The dense point clouds could be edited andclassified or exported to an external tool for further analysis. Then, the classified dense point clouds ofthe plant were transformed into polygon meshes. This procedure did some basic filtering to eliminate
the most obviously irrelevant points. Thereafter, such polygon wireframe images of the plant allowed
calculating the distance or area of leaves and stems.
Therefore, the 3D surface texture of leaves and stems was expressed, as the inconsistent
triangle meshes depend especially on the alignment of the point cloud after noise exclusion.
Moreover, triangulated meshing and surface reconstruction were applied to generate surface texture
models. Laplacian smoothing was employed to smooth triangle meshing. A series of natural color
photos captured after wireframing were matched on the surface model of the entire plant using a
texture-mapping approach. The texture-mapping approach is generally used as a method for putting
surface texture into a computer-reconstructed 3D surface model [54]. The 2D photos like natural color
photos are available for the surface model reconstruction. Through the coordinate’s correspondenceof the 2D image to the 3D model, the feature of the 2D image is automatically matched onto each
wireframe shell of the 3D model. A rigid choice of the matching points between 2D and 3D images
is very important to generate the complicated texture mapping of the paprika. Then, the matching
points were selected from unusual points on each stem and leaf; whereafter, exact 3D natural color
models were constructed. From the models, some information can be refined, such as plant diameterand height, leaf shapes and area, which are significant in plant detection and recognition.
2.5. 3D Model Measurement of Nursery Plants
In order to estimate the parameters of the plant from the 3D model, we choose the twoclearly-distinguished Rubik’s cubes on two sides of the nursery plant box as references to set thereferenced coordinates’ points because of their standard 3D structure. The length of each edge has afixed value of 66 mm. The coordinates in one of the two cubes were set as reference coordinates withthree-dimensional coordinates (X, Y, Z). These georeferenced point clouds can be used to provide very
specific and exact representations of a close depth scene. The measurement parameters include leaf
width, leaf length, stem diameter and stem height. According to the coordinates of the referenced
points in the 3D model, they were estimated by calculating the distances of the measured marked
points in each leaf of the 3D model. For the measurement of each leaf, the distance between the upper
end of the petiole to the leaf apex was calculated as the leaf length along the leaf vein. The distance fromone edge to another edge in the middle part of the blade was calculated as the leaf width. As for themeasurement of the stem, the stem diameter was calculated in the center of the stem. Stem height wascalculated from stem tip to soil surface. In this experiment, 18 leaves and 9 stems were chosen fromthe 3D model and about 100 points on the 3D model were marked for the measurement in each leaf
and stem, as well as the cube reference. For the direct measurement of plant growth parameters, we
adopted the destructive method to measure the leaf, because plant leaves have irregular shapes anddifferent states, such as wide or narrow, straight or curly, etc. In the experiment sample pot, there wereSensors 2016, 16, 874

8 of 18

9 paprika plants in total. We selected two sample leaves from each paprika plant. We cut and measured
them by a ruler. The measurement method for every leaf and stem is the same as the above. Finally,
from leaf width and length to stem diameter and height, including spatial summaries of nursery plants,
we could obtain the direct and indirect measurement data. According to these data, we could calculate
correlation coefficients and Root-Mean-Square Errors (RMSE) between the estimated and measured
values of these parameters to obtain their accuracy. We were also able to obtain the percentage of thetotal number of virtual leaves and stems whose 3D model surface area reconstructed over 95% or 75%,
accounting for the total number of real leaves and stems, to compare the performance of different lenses.
3. Results
3.1. 3D Modeling Reconstruction of Paprika Plant
Based on the MSV and SfM algorithms, we have applied the high efficiency MCP-SfM system
to obtain many high resolution images from close-up views of four positions. We have respectively
tested four types of lenses for one circle of plant shape in four different camera positions, whose focal
lengths were 24 mm, 28 mm, 35 mm and 50 mm. Table 1 shows that the numbers of acquired imagesof different lenses in the first and second camera positions, which were on average about 42 photos.
However, for the third and fourth camera positions, the number of acquired images of the 50-mm lensis the smallest, because this lens is more sensitive to light intensity in the third and fourth camera positions.
Because the MCP-SfM system has been set with certain shooting distance and the indoor light condition,
Table 1 demonstrates that the 28-mm lens had good performance while taking photos of nursery plants.
Table 1. The number of images taken by different lenses.

Camera Positions
1
2
3
4
Total

Focal Length of Lens
24 mm

28 mm

35 mm

50 mm

44
42
42
42
170

44
41
43
45
173

44
44
42
43
173

40
42
29
23
134

In terms of the resolution and performance of different lenses, we found that 28 mm was better
than other focal lengths. The features of the paprika SfM model are shown in Figure 4.
Figure 4 showed the procedure of 3D modeling for nursery paprika plants reconstructed from aseries of photos taken by the 28-mm lens at the first camera position. Firstly, according to the process of
3D modeling, PhotoScan generated point clouds with a total of 88,022 points after registration from the
44 input images captured by the camera, as shown in Figure 4A. The number of points per leaf ranged
from 105 to 5140, depending on leaf size. Currently, each point in the clouds of points is determinedby two or more images, and overlapping areas have a lot of points. It would be more accurate if thiswere determined by all possible images, with fewer points, but more precision in the points. It is best
to define the minimum number of images for the determination of a point. Secondly, after removing
the noise within the point cloud dataset, a total of 1,528,730 dense points remained in Figure 4B,of which 1,528,730 points were classified as shown in gray in Figure 4C. The dense point clouds gave
greater precision, and the point number is thirty times more than one of the point clouds and much
more accurate. Then, the classified dense clouds of plant leaf were converted into the corresponding
polygon wireframe in Figure 4D. It was seen that the polygons consist of many inconsistent triangle
wireframes. The complex curves and unevenness on the paprika plant model were generated on thepolygon wireframes. The number of polygon wireframes per leaf ranged from 139 to 5765, which was
determined by leaf size.

3
4
Total

42
42
170

43
45
173

42
43
173

29
23
134

In terms
Sensors 2016,
16, 874of the resolution and performance of different lenses, we found that 28 mm was better
9 of 18
than other focal lengths. The features of the paprika SfM model are shown in Figure 4.

Sensors 2016, 16, 874

9 of 17of 3D modeling, PhotoScan generated point clouds with a total of 88,022 points after registration fromthe 44 input images captured by the camera, as shown in Figure 4A. The number of points per leaf
ranged from 105 to 5140, depending on leaf size. Currently, each point in the clouds of points isdetermined by two or more images, and overlapping areas have a lot of points. It would be moreaccurate if this were determined by all possible images, with fewer points, but more precision in the
points. It is best to define the minimum number of images for the determination of a point. Secondly,
after removing the noise within the point cloud dataset, a total of 1,528,730 dense points remained in
Figure 4B, of which 1,528,730 points were classified as shown in gray in Figure 4C. The dense point
Figure 4.
Procedureofof3D
3Dmodelingmodelingfor
forpaprika
paprika plants
plants reconstructed fromaaseries
ofof
photostaken
byby
FigureProcedure
fromthanseriesphotostaken
clouds 4.
gave
greater precision,and the point
number is reconstructedthirty times more
one ofthepointclouds
the
28-mm
lens
at
thefirst
firstcamera
cameraposition.
position. (A)(A) Point
Point cloud;
(B)
Dense
cloud;
(C)
Dense
cloud
classes;
the
28-mm
lens
at
the
cloud;(B)
Dense
cloud;
(C)
Dense
cloud
classes;and much more accurate. Then, the classified dense clouds of plant leaf were converted into the(D)
Wireframe.
(D)
Wireframe.
corresponding
polygon wireframe in Figure 4D. It was seen that the polygons consist of many
inconsistent triangle wireframes. The complex curves and unevenness on the paprika plant model
Figure 4 while
showedthe procedureof 3Dmodeling
foralsonursery
paprikawith
plants reconstructedfromaTypically,
thewireframes.
point
clouds,noise
increased
pointfromdensity.
were generated
onbuilding
the polygon
The numberof polygon
wireframesincreasing
per leaf ranged
seriesofphotos
taken
by
the
28-mm
lens
at
the
first
cameraposition.
Firstly,accordingtotheprocess
The noise
were wasa little
higher or
than that expected, which cannot reproduce any actual
139 topoints
5765, which
determined
bylowerleaf size.
characteristics
withinthebuilding
paprika
Besides,theextentof thepoint
cloud point
datasetwas also
Typically,
while
theplants.
point clouds,
noise
also
increasedwithincreasing
density.
The noisepoints wereprocessing
a little higher
or lowerthat expected,
cannotreproduce
actualdecreased
to minimize
time,
and thanthe point
clouds which
outsideof theregion any
were
directly
characteristics
withinthepaprikaplants.
Besides,theextent
of
the
point
cloud
datasetwas
alsoclassified as noise points. Most of these were removed during building the dense clouds. The filtered
to minimize
processing
and theclouds plants,
outside asof shown
the region
were directly
pointdecreased
clouds produced
an accurate
3Dtime,modelingof point
the paprikain Figure
4. Each leaf
classified as noise points. Most of these were removed during building the dense clouds. The filteredor stem feature was clear because of the high-resolution photos captured by the system.
point clouds produced an accurate 3D modeling of the paprika plants, as shown in Figure 4. Each
Figure 5 shows the texture mapping of the best 3D models with 0.25-mm resolution for paprika
leaf or stem feature was clear because of the high-resolution photos captured by the system.
plants by aFigure
28-mm
lens in the first and third camera position. In Figure 5A, leaf models are reconstructed
5 shows the texture mapping of the best 3D models with 0.25-mm resolution for paprika
from plants
a seriesbyofaphotos
by thea 28-mmlensthird
at thefirst camera
position.
In Figureleaf and28-mmtaken
lens infirst andcameraposition.
In Figure
5A, leaf5B,
modelsare stemmodelswerereconstructedfromaseriesofphotos
taken
by
a
28-mm
lens
at
the
third
cameraposition.
reconstructed from a series of photos taken by a 28-mm lens at the first camera position. In Figure 5B,
Comparing
5A towere
Figure5B, the firsthigherresolution.
leaf andFigure
stem models
reconstructedfromcamera
a series positionof photos hadtakena by
a 28-mmlens at theMoreover,thirdcameraFigure
5A toinFigure5B,pots
the first
position
hadthana higher
Figure5 alsoposition.
shows Comparingthat paprika
modelsouter
are camera
generally
betteronesresolution.
in inner pots
Figure5 alsoshowsFurthermore,that paprika models
in outerpotsofarenursery
generally
betterthan ones
in beby theMoreover,
28-mm lensin thedetails.
even lower
parts
plant
shapescan also
inner
pots
by
the
28-mm
lens
in
the
details.
Furthermore,
even
lower
parts
of
nurseryplantshapesestimated and reconstructed. In sum, from Figure 5, we can see that most of the leaf and stem shapes
also be estimated and reconstructed. In sum, from Figure 5, we can see that most of the leaf andwere canreconstructed,
and their clear models were generated by the 28-mm lens in the first and third
stem shapes were reconstructed, and their clear models were generated by the 28-mm lens in the firstcamera positions, respectively.
and third camera positions, respectively.

Figure5. Texture
mappingof thebest
3Dmodelsmodels forfor paprika
paprika plants.
Leafmodelsreconstructed
Figure5. Texture
mappingof thebest
3D
plants.(A)(A)
Leafmodelsreconstructedfrom a series of photos taken by a 28-mm lens at the first camera position; (B) Leaf and stem models
from a series of photos taken by a 28-mm lens at the first camera position; (B) Leaf and stem models
reconstructed from a series of photos taken by a 28-mm lens at the third camera position.
reconstructed from a series of photos taken by a 28-mm lens at the third camera position.

3.2. Parameters’ Estimation of the 3D Paprika Plant Model
According to the set georeferenced coordinate system and created measurement points in the
3D model, the estimation for leaf and stem shape parameters was conducted. For example, in
Figure 5A, the generated leaf lengths ranged between 31 mm and 64 mm over all paprika plants.

Sensors 2016, 16, 874

10 of 18

3.2. Parameters’ Estimation of the 3D Paprika Plant Model
According to the set georeferenced coordinate system and created measurement points in the 3Dmodel, the estimation for leaf and stem shape parameters was conducted. For example, in Figure 5A,
the generated leaf lengths ranged between 31 mm and 64 mm over all paprika plants. Likewise,in Figure 5B, the generated stem heights ranged between 53 mm and 96 mm. The difference of ninepaprika stems’ height between the measured height and the estimated height was found to be lessthan 1.14 mm. The difference of nine paprika stems’ diameter between the measured value and theestimated value was under 0.76 mm. The average difference of stem height and diameter was about
0.6 mm and 0.31 mm, respectively. Figure 5 demonstrates that the 3D model reconstructed by the
28-mm lens in the first and third camera positions is better than other lenses in other camera positions
and camera position groups.
Table 2 illustrates the percentage of the total number of virtual leaves and stems whose 3D model
shape surface area reconstructed over 95% (Category A) or 75% (Category B) accounting for the total
number of real leaves and stems. Ninety five percent and 75% mean the percentage of the reconstructedsurface area of each leaf and stem shape compared to the real shape in the 3D model. Category A
means the percentage of the 3D model shape surface area reconstructed over 95% compared to thereal shape for both leaves and stems. Category B means the percentage of the 3D model shape surface
area reconstructed over 75% compared to the real shape for both leaves and stems. The values inTable 2 were obtained by calculating the total number of virtual leaves and stems whose 3D model
shape surface area reconstructed over 95% or 75% divided by the total number of real leaves andstems in the pot. For example, the total number of real plant leaves is 74. For the 28-mm lens in
the first camera position, the total number of virtual leaves whose 3D leaf model shape surface area
reconstructed over 95% is 35. Therefore, by dividing 35 by 74, the approximate value of 0.4729 can beobtained. Hence, the percentage of the total number of virtual leaves whose 3D model shape surface
area reconstructed over 95% accounting for the total number of real leaves is about 47%, as seen in
Figure 5A. Other values for leaves in Table 2 were obtained in the same way. In other camera positions
or camera position groups, the percentage of the total number of virtual leaves whose 3D model shapesurface area reconstructed over 95% accounting for the total number of real leaves is mostly less than
40%. Likewise, the values for stems in Table 2 can also be obtained in the same way. As can be seen in
Table 2, for the 28-mm lens in the third camera position, the percentage of the total number of virtual
stems whose 3D model shape surface area reconstructed over 95% accounting for the total numberof real leaves is about 100%. Therefore, all of the stem shapes were reconstructed by the 28-mm lensin the third camera position, as seen in Figure 5B. However, in other camera positions or camera
position groups, the percentage of the total number of virtual leaves whose 3D model shape surface
area reconstructed over 95% accounting for the total number of real leaves is mostly less than 90%.
Thus, from Table 2, we can see that the 28-mm lens in the first and third camera position has great
performance in reconstructing the 3D model of the plant.
In general, according to the parameters’ estimation result of the 3D paprika plant model, we cananalyze the performance of different lens types in different camera positions. These analyses are very
helpful for understanding the monitoring of plant growth parameters and the environment, such as
light conditions and disease detection.

Sensors 2016, 16, 874

11 of 18

Table 2. Percentage of the total number of virtual leaves and stems whose 3D model shape surface area reconstructed over 95% (Category A) or 75% (Category B)
accounting for the total number of real leaves and stems.
Percentage of the Number of Category A or B to That of All Leaves (%) *1
Camera Positions(see Figure 1)

Percentage of the Number of Category A or B to That of All Stems (%) *2

A *3

B *4

A *3

B *4

>95

>75

>95

>75

Focal Length of Lens (mm)

1
2
1+2
3
1+2+3
4
1+2+3+4

24

28

35

50

24

28

35

50

24

28

35

50

24

28

35

50

34
20
39
3
9
4
7

47
27
45
5
26
5
14

31
27
28
1
27
4
9

32
22
22
0
0
3
16

57
54
59
11
49
14
42

76
59
68
24
50
19
46

74
64
66
9
69
22
41

62
68
64
5
0
5
68

0
44
56
78
78
44
89

0
57
78
100
89
56
89

0
44
44
100
33
44
22

0
33
33
33
0
22
33

11
67
67
89
89
56
89

11
78
78
100
89
56
100

11
78
56
100
56
78
44

0
67
56
44
0
44
44

*1 The total number of real plant leaves is 74; *2 The total number of real plant stems is 9; *3 Category A means the percentage of 3D model shape surface area reconstructed over 95%
compared to the real shape; *4 Category B means the percentage of 3D model shape surface area reconstructed over 75% compared to the real shape.

Sensors 2016, 16, 874

12 of 18

Sensors 2016, 16, 874

12 of 173.3.Error
ErrorMeasurement
Measurement ofof thethe 3D
3D Paprika
Paprika Plant
Plant Model
3.3.
Model
In ourour experiment,
experiment, thethe error
error analyses
analyses ofInof thethe estimatedestimated parameters,
parameters, such
suchasas thethecorrelation
correlation
coefficient
and
RMSE,
were
conducted
fordifferent
lens
types
and
camerapositions.
Accordingtocoefficient and RMSE, were conducted for different lens types and camera positions.
Accordingtheerror
analyses’
comparison,
the resultsof thelens were
to be the
Some
to
the
error
analyses’
comparison,
the resultsof28-mmthe 28-mm
lens demonstrated
were demonstrated
to best.
be thebest.
modelsgeneratedby
otherlenses
are
not
so
good,
with
more
noise
and
calibration
errorsin
someinSome models generated by other lenses are not so good, with more noise and calibration errors
parts partsof theof3D
For example,
the absolute
error ofthe ofNo.
paprikastem heightfor ninesomethemodel.
3D model.
For example,
the absoluteerror
the1 No.
1 paprika
stem heightfor
paprika
stems
is
0.792
mm.
Its
relativeerror
is0.008,
whichisobtainedfrom
0.792
dividedby
99
nine paprika stems is 0.792 mm. Its relative error is 0.008, which is obtained from 0.792 divided by(measuredvalue).
The
absolute
error
of height
forfornine
paprikastems
is under0.2 mm.
99(measuredvalue).
The
absolute
error
of height
nine
paprikastems
is under0.2 mm.
2) and the regression results of leaf and stem
Figure6showsthecoefficientofdetermination(R
Figure 6 shows the coefficient of determination (R2 ) and the regression results of leaf and stem forfor the 28-mm lens. The measured and estimated values were showed in a one-to-one relationshipthe 28-mm lens. The measured and estimated values were showed in a one-to-one relationship andand a regression fit trend line. This demonstrated the accuracy of the best 3D models for leaf anda regression fit trend line. This demonstrated the accuracy of the best 3D models for leaf and stem
stem shapes of paprika plants taken by the 28-mm lens. In Figure 6A, for the leaf length and width ofshapes of paprika plants taken by the 28-mm lens. In Figure 6A, for the leaf length and width of plants
plants taken at the first camera position, the correlation coefficient between the measured andtaken at the first camera position, the correlation coefficient between the measured and estimatedestimated values showed a good fit. Likewise, in Figure 6B, for stem heights and the diameters ofvalues showed a good fit. Likewise, in Figure 6B, for stem heights and the diameters of plants taken atplants taken at the third camera position, the regression fit between the measured and estimatedthe third camera position, the regression
fit between the measured and estimated values was higher,
values was higher, giving an R2 of approximately 1.0. Besides, the spatial resolution was 0.25 mm per
giving an R2 of approximately 1.0. Besides, the spatial resolution was 0.25 mm per pixel on the Y-axis
pixel on the Y-axis and 0.329 mm per pixel on the Z-axis. The accuracy assessment is high with aand 0.329 mm per pixel on the Z-axis. The accuracy assessment is high with a mean difference of themean difference of the Z-values of 0.6 mm. Therefore, the accuracy of the 3D model by our MCP
Z-values of 0.6 mm. Therefore, the accuracy of the 3D model by our MCP system is reliable.
system is reliable.

Figure 6. The accuracy of the best 3D models for leaf and stem shapes of paprika plants taken by the
Figure 6. The accuracy of the best 3D models for leaf and stem shapes of paprika plants taken by the
28-mm lens. (A) Leaf length and width of plants taken at the first camera position; (B) Stem heights
28-mm lens. (A) Leaf length and width of plants taken at the first camera position; (B) Stem heights
and diameters of plants taken at the third camera position.
and diameters of plants taken at the third camera position.

In addition, the RSME of the errors was computed along with the regression fit degree and R2 2In
addition,
theandRSMEof the errors
was computedwith
the3,regression
fit degree
and3DR
betweenmeasuredestimatedparameters.
As can bealong
seen inTable
the error resultsof thebetweenmeasuredandestimatedparameters.
As
can
be
seen
in
Table
3,
the
error
resultsofthe3D
2
model of best leaf and stem taken by the 28-mm lens are illustrated. The R and RMSE of paprika leaf
modelof bestleaf
and stemby theexperiment
28-mm lensaccording
are illustrated.
R2 and RMSE
of paprika
and stem
were
presented
fortaken
the whole
to the The
measurement
technique
and
leaf
and
stem
were
presented
for
the
whole
experimentaccordingtothemeasurement
techniqueandmethod. In the first camera position, it showed a correlation coefficient of 0.9821 (n = 18) between themethod.
In
the
first
cameraposition,itshowedacorrelation
coefficientof0.9821(n=
18)
betweenthemeasured and estimated values with a mean difference of 1.65 mm. Likewise, in the third camera
measuredandRMSEestimatedvaluesa meanmm. Likewise,
in of
the0.9998
third (ncameraposition, theof stem
was with
1.65 mm,anddifference
it showedofa 1.65
correlation
coefficient= 9)
position,
the
RMSE
of
stem
was
1.65
mm,
and
it
showedacorrelation
coefficientof
0.9998(n
between the measured and estimated values with a mean difference of 0.57 mm. Besides, Table=39)
betweenthe measured
estimated forvaluesdifferenceof 0.57mm.
Besides,Tableshows better
accuracy andof estimation
stemwiththana mean
for leafin the caseof the
28-mm
lens.
The 3
shows
betterdifference
accuracy of
estimation
for stemleaf
in thecase oflens. ofThe
estimation
estimation
between
paprikastemthanandforleaf
wasbecauseofthe
the28-mm
limitations
theremotedifference
between
paprika
stem
and
leaf
was
becauseofthelimitationsoftheremotesensing
method.
sensing method.
All
in
all,
the
above
results
demonstratedageneralconsistency
betweentheestimatedandAll in all, the above results demonstrated general consistency between the estimated andmeasured
forfor
3D plantmodelmodelreconstruction
by analyzing
the statistically-significant
differences
measuredvalues
values3D plant
reconstruction
by analyzing
the statistically-significant
among
different
lens
types, lenspointing
the promisingrole of thewith several
differencesamong
different
types,outpointing
out the promisingroleMCPof thesystem
MCP systemwithconsumer-grade
camerascameras
for the purposesof plantof
growthmonitoring.
several consumer-grade
for the purposes
plant growthmonitoring.

Sensors 2016, 16, 874

13 of 18

Table 3. The error results of the 3D model of the best leaf and stem taken by the 28-mm lens.

Camera Positions
1
3

Leaf

Stem

R2

RMSE (mm)0.9821

1.65

R2

RMSE (mm)0.99980.57

4. Discussions and Conclusions
4.1. Advantages of the MCP System Based on the MVS and SfM Methods
This study presented an evaluation of the accuracy of the 3D image model derived by the
MCP-SfM system based on the MVS and SfM technologies. The techniques are usually based on theconsistency measurement, where the 3D model is consistent with the input images. They provide
viable methods for generating 3D models of vegetation and scenes. The MVS method is a common
application of 3D modeling of nursery plants. The SfM method is used to generate 3D models fortriangulated meshing and surface reconstruction. These models generated by different lenses or camera
positions were compared and evaluated. In general, the SfM method has a great potential in remotesensing range data capture of natural and man-made landscapes to provide very detailed and precise
representations. Some small experiments indoors, like the one presented in this study, can be easily
monitored using the system. Producing multi-temporal datasets of the entire growth period is essential
for obtaining reliable results in such experiments. In the present study, we measured the performanceof the highly effective and low cost systems with digital cameras and image reconstruction techniques
for estimating the four variables of paprika plants, such as leaf width and length and stem heightand diameter. The results showed the practicability of our new method for an exact measurement ofplant parameters. The system can make the construction and acquisition of images faster and generatesurface models from more viewing angles. The system is also available for extension into real-timemonitoring of plant growth in the future. Therefore, it is an ideal system platform for taking more highresolution 3D images of plants.
4.2. Error Assessment of the 3D Paprika Plant Model
For the MCP-SfM system, the challenge is the performance and accuracy of the SfM algorithm
and the accuracy of 3D depth data with the effect of ambient illumination. In fact, apart from the error
analysis of the 3D paprika plant model, the accuracy of depth data is usually divided between theabsolute and relative errors. The absolute error is obtained by the comparison with a known reference
value. In the presented case, the absolute error is obtained by comparing the estimated value to ameasured one. The relative error can be estimated by calculating the absolute error (the difference
between the estimated value and the measured value) divided by the measured value. It gives anindication of how good a measurement is relative to the size of the object being measured. Therefore,
the comparative accuracy of these measurements can be determined by looking at their relative
errors. Besides, the relative error in the parameters’ measurement can better reflect the reliability ofmeasurement accuracy. In particular, this could be considered as one of the reliable evaluation tools in
future Digital Elevation Model (DEM) studies.
The precisions achieved on the individual paprika displayed variations determined by the typeof the lens and the position of the camera. According to our various experimental comparisons,a focal length larger than 35 mm could not better reconstruct high resolution 3D models because
the measurement devices were installed in a limited scope for indoor conditions. Besides, the upper
camera positions were better for the measurement purposes of leaf shape in the 3D model, whereas thelower camera positions are better for the measurement purposes of stem shape. The camera positiongroups were better for the measurement purposes of the whole plant shape in the 3D model. In general,the performance was the best using the 28-mm lens in the first and third camera positions. AccordingSensors 2016, 16, 874

14 of 18

to Table 3, the RMSE result of these errors was considered an acceptable error range for estimating
the features in consideration of the possible errors’ magnitude during manual measurements of theparameters, which could give the validity of the reference data [55,56]. Errors were slightly higher forother lenses or camera positions. What is more, in the above studies, we also found low correlations
between the estimated and measured variables, which showed the influence of some limitations of
this condition.
In addition, according to the percentage of the total number of virtual leaves and stems whose
3D model shape surface area reconstructed over 95% or 75% accounting for the total number of realleaves and stems (Table 2), the errors for the reconstructed 3D model are obvious among different
focal lengths and camera positions. The main reasons for the errors in the parameters’ measurement
might be not only different overlap rates due to different captured image numbers, but also different
exposure rates due to some particularities of the ambient illumination condition. Another reason is the
distance between paprika plants and the camera lens in the indoor experimental site, because somelarger focal lengths need a longer distance, such as 50 mm and 85 mm. Thus, it is important for higher
3D model accuracy to adjust the suitable ambient illumination condition and shooting distance in thelimited lab scope based on the lighting conditions of different camera positions and the characteristics
of different lenses.
4.3. Limitations and Issues of the Experiment
In the construction of point clouds, the noises faced in some areas with a complex structure
result in sparse black patches, which need to be well represented. According to our investigation,
there are some limitations of the imaging system combined with environmental factors. One of themost important factors is light. Generally, lighting should be diffused to reduce errors. Under indoor
conditions, the background and shading arrangements have been used to cater to the experiments.
In this way, the effect of ambient illumination for the camera varies with the type of lens used.
As the quantity, resolution and luminosity of images captured from the cameras of different views aredifferent, there are some differences among different 3D models. In this study, we have discussed theexposure of the camera for the 3D imaging system, which is robust to variation in ambient illumination.
It is necessary to compare and discuss the effect of lighting variations under different conditions
on the plant canopies in the future. Besides, according to the different quantity of images and theperformance of different computers, the entire workflow for producing 3D models from MCP-SfM
images requires different time. For stereo images with dimensions of 5184 ˆ 3456, it is important toarrange a uniform-luminance environment and a high-performance computer. Keeping the above
limitations aside, it is also necessary for further research in the close range stereo imaging of plants toparticularly take into account color distortion due to the lighting, sensors and exposure control, as
discussed by Nielsen et al. [57]. Moreover, there are inherent limitations of the stereo matching process
in this system, which is not robust to all sorts of surfaces and objects expected in agricultural scenarios.
This reduces the effect of stereovision and limits either the range or the scale of the application. Finally,other issues with the system are the high performance requirement for the CPU and the large memory
needed, especially when we try to do 3D reconstruction at a large scale.
4.4. Future Improvements and Applications
According to the above-mentioned issues, some improvements should be made in the future.
In order to ensure that the camera in nadir position can take high quality images while capturing
images, an adjustable light needs to be fixed for increased resolution. Additionally, we will further
propose a method for obtaining the most suitable camera configurations for 3D imaging under different
illumination conditions based on observing the precision trends in the camera and lens. Besides, anelectrical trigger will also be used to make image acquisition more reliable compared to a mechanicalshutter. Meanwhile, taking other sources of error into account, like the inaccuracies caused by rotating
plants during data acquisition, the overlap of 90% among the images will be increased for higherSensors 2016, 16, 874

15 of 18

accuracy in the future. Different settings were tested in PhotoScan showing that the model quality
increased with the number of photos used for model generation. This is also stated by Roberts et al. [58].
Thus, in order to enable plant growth monitoring with higher accuracy, the number of photos taken
during the illumination will be increased with a greater variety of viewing perspectives in the future.
According to our next design, six more cameras will be installed in the system. Moreover, in the future,
the models will also be expected to detect the differences between growths according to the cultivar
and treatment.
Acknowledgments: We gratefully acknowledge Parinaz Rahimzadeh-Bajgiran and Hayashi Masao for theirkind support.
Author Contributions: K.O. and Y.Z. conceived and designed the experiments; Y.Z. performed the experiments,
analyzed the data and wrote the paper; K.O., P.T., F.H., and Y.S. contributed material, analysis tools, discussions
and paper revision.
Conflicts of Interest: The authors declare no conflict of interest.

Abbreviations
The following abbreviations are used in this manuscript:
MCP
MVS
SfM
RMSE
SIFT
PPF
PnP
SSD
DEM

Multi-Camera Photography
Multiple-View Stereovision
Structure from Motion
Root-Mean-Square Error
Scale-Invariant Feature Transform
Photosynthesis Photon Flux
Perspective N-Point
Sum of Squared Differences
Digital Elevation Model

References
1.

2.
3.
4.

5.
6.

7.

8.

9.

Muller-Linow, M.; Pinto-Espinosa, F.; Scharr, H.; Rascher, U. The leaf angle distribution of natural plant
populations: Assessing the canopy with a novel software tool. Plant Methods 2015, 11, 1–16. [CrossRef][PubMed]Omasa, K.; Hosoi, F.; Konishi, A. 3D lidar imaging for detecting and understanding plant responses andcanopy structure. J. Exp. Bot. 2007, 58, 881–898. [CrossRef] [PubMed]
Spalding, E.P.; Miller, N.D. Image analysis is driving a renaissance in growth measurement. Plant Biol. 2013,16, 100–104. [CrossRef] [PubMed]
Omasa, K.; Takayama, K. Simultaneous measurement of stomatal conductance, non-photochemical
quenching, and photochemical yield of photosystem II in intact leaves by yhermal and chlorophyll
fluorescence imaging. Plant Cell Physiol. 2003, 44, 1290–1300. [CrossRef] [PubMed]
Omasa, K. Image Analysis of Chlorophyll Fluorescence Transients of Cultured Carrot Tissues.
Environ. Control Biol. 1992, 30, 127–131. [CrossRef]
Omasa, K. Image Instrumentation Methods of Plant Analysis. In Modern Methods of Plant Analysis;
Linskens, H.F., Jackson, J.F., Eds.; New Series, Physical Methods in Plant Sciences; Springer-Verlag: Berlin,
Germany, 1990; Volume 11, pp. 203–243.
Omasa, K.; Takayama, K.; Goto, E. Image Diagnosis of Photosynthetic Injuries Induced by Herbicidein Plants—Comparison of the Induction Method with the Saturation Pulse Method for Chlorophyll a
Fluorescence Analysis. J. Soc. High Technol. Agric. 2001, 13, 29–37. [CrossRef]
Barbagallo, R.P.; Oxborough, K.; Pallett, K.E.; Baker, N.R. Rapid, noninvasive screening for perturbationsof metabolism and plant growth using chlorophyll fluorescence imaging. Plant Physiol. 2003, 132, 485–493.
[CrossRef] [PubMed]
Garcia, R.; Francisco, J.; Sankaran, S.; Maja, J. Comparison of two aerial imaging platforms for identificationof Huanglongbing-infected citrus trees. Comput. Electron. Agric. 2013, 91, 106–115. [CrossRef]

Sensors 2016, 16, 874

10.

11.
12.
13.
14.

15.
16.

17.
18.

19.
20.
21.
22.
23.

24.
25.
26.

27.
28.
29.
30.

31.

16 of 18

Nuske, S.; Achar, S.; Bates, T.; Narasimhan, S.; Singh, S. Yield estimation in vineyards by visual grape
detection. In Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems,
San Francisco, CA, USA, 25–30 September 2011; pp. 2352–2358.
Astrand, B.; Baerveldt, A. Plant recognition and localization using context information. In Proceedings ofthe IEEE Conference Mechatronics and Robotics, Aachen, Germany, 13–15 September 2004.
Li, L.; Zhang, Q.; Huang, D. A Review of Imaging Techniques for Plant Phenotyping. Sensors 2014, 14,20078–20111. [CrossRef] [PubMed]
Lee, W.S. Sensing technologies for precision specialty crop production. Comput. Electr. Agric. 2010, 74, 2–33.
[CrossRef]
Preuksakarn, C.; Boudon, F.; Ferrao, F.; Durand, J.-B.; Nikinmaa, E.; Godin, C. Reconstructing plantarchitecture from 3D laser scanner data. In Proceedings of the 6th International Workshop on
Functional-Structural Plant Models, Davis, CA, USA, 12–17 September 2010; pp. 1999–2001.
Takizawa, H.; Ezaki, N.; Mizuno, S.; Yamamoto, S. Plant Recognition by Integrating Color and Range Data
Obtained Through Stereo Vision. J. Adv. Comput. Intell. Intelli. Inform. 2005, 9, 630–636.
Hosoi, F.; Omasa, K. Estimating vertical plant area density profile and growth parameters of a wheat canopy
at different growth stages using three-dimensional portable lidar imaging. ISPRS J. Photogramm. Remote Sens.
2009, 64, 151–158. [CrossRef]
Hosoi, F.; Nakabayashi, K.; Omasa, K. 3-D Modeling of tomato canopies using a high-resolution portablescanning lidar for extracting structural information. Sensors 2011, 11, 2166–2174. [CrossRef] [PubMed]
Nielsen, M.; Andersen, H.; Granum, E. Comparative Study of Disparity Estimations with Multi-Camera
Configurations in Relation to Descriptive Parameters of Complex Biological Objects. In Proceedings of the
ISPRS Workshop BenCOS 2005: Towards Benchmarking Automated Calibration, Orientation and Surface
Reconstruction from Images, Beijing, China, 15 October 2005.
Hosoi, F.; Omasa, K. Estimating leaf inclination angle distribution of broad-leaved trees in each part of the
canopies by a high-resolution portable scanning lidar. J. Agric. Meteorol. 2015, 71, 136–141. [CrossRef]
Endres, F.; Hess, J.; Sturm, J.; Cremers, D.; Burgard, W. 3D Mapping with an RGB-D Camera.
IEEE Trans. Robot. 2014, 30, 177–187. [CrossRef]
Khoshelham, K.; Elberink, S.O. Accuracy and resolution of kinect depth data for indoor mapping applications.
Sensors 2012, 12, 1437–1454. [CrossRef] [PubMed]
Omasa, K.; Kouda, M.; Ohtani, Y. 3D microscopic measurement of seedlings using a shape-from-focus
method. Trans. Soc. Instrum. Control Eng. 1997, 33, 752–758. [CrossRef]
Mizuno, S.; Noda, K.; Ezaki, N.; Takizawa, H.; Yamamoto, S. Detection of Wilt by Analyzing Color andStereo Vision Data of Plant. In Computer Vision/Computer Graphics Collaboration Techniques of Lecture Notes
in Computer Science; Gagalowicz, A., Philips, W., Eds.; Springer Berlin Heidelberg: Berlin, Germany, 2007;
Volume 4418, pp. 400–411.
Scharstein, D.; Szeliski, R. A Taxonomy and Evaluation of Dense Two-Frame Stereo Correspondence
Algorithms. Int. J. Comput. Vis. 2002, 47, 7–42. [CrossRef]
Kyto, M.; Nuutinen, M.; Oittinen, P. Method for measuring stereo camera depth accuracy based onstereoscopic vision. Proc. SPIE 2011, 7864. [CrossRef]
Van der Heijden, G.; Song, Y.; Horgan, G.; Polder, G.; Dieleman, A.; Bink, M.; Palloix, A.; van Eeuwijk, F.;
Glasbey, C. SPICY: Towards automated phenotyping of large pepper plants in the greenhouse.
Funct. Plant Biol. 2012, 39, 870–877. [CrossRef]
Biskup, B.; Scharr, H.; Schurr, U.; Rascher, U. A stereo imaging system for measuring structural parametersof plant canopies. Plant Cell Environ. 2007, 30, 1299–1308. [CrossRef] [PubMed]
Kise, M.; Zhang, Q. Development of a stereovision sensing system for 3D crop row structure mapping andtractor guidance. Biosyst. Eng. 2008, 101, 191–198. [CrossRef]
Ivanov, N.; Boissard, P.; Chapron, M.; Valery, P. Estimation of the height and angles of orientation of theupper leaves in the maize canopy using stereovision. Agronomie 1994, 14, 183–194. [CrossRef]
Kazmi, W.; Foix, S.; Alenya, G.; Andersen, H.J. Indoor and outdoor depth imaging of leaves with time offlight and stereo vision sensors: Analysis and comparison. ISPRS J. Photogramm. Remote Sens. 2014, 88,
128–146. [CrossRef]
Okutomi, M.; Kanade, T. A multiple-baseline stereo. IEEE Trans. Pattern Anal. Mach. Intell. 1993, 15, 353–363.
[CrossRef]

Sensors 2016, 16, 874

32.

33.
34.
35.

36.
37.
38.

39.
40.
41.

42.

43.
44.
45.

46.
47.
48.

49.

50.
51.
52.

53.

17 of 18

Vogiatzis, G.; Hernández, C.; Torr, P.H.S.; Cipolla, R. Multiview stereo via volumetric graph-cuts andocclusion robust photoconsistency. IEEE Trans. Pattern Anal. Mach. Intell. 2007, 29, 2241–2246. [CrossRef][PubMed]Vu, H.-H.; Labatut, P.; Keriven, R.; Pons, J.P. High accuracy and visibility-consistent dense multi-view stereo.
IEEE Trans. Pattern Anal. Mach. Intell. 2012, 34, 889–901.
Harwin, S.; Lucieer, A. Assessing the Accuracy of Georeferenced Point Clouds Produced via Multi-View
Stereopsis from Unmanned Aerial Vehicle (UAV) Imagery. Remote Sens. 2012, 4, 1573–1599. [CrossRef]
Fonstad, M.A.; Dietrich, J.T.; Courville, B.C.; Jensen, J.L.; Carbonneau, P.E. Topographic structure frommotion: A new development in photogrammetric measurement. Earth Surf. Proc. Landf. 2013, 38, 421–430.
[CrossRef]
Lowe, D. Distinctive image features from scale-invariant key points. Int. J. Comput. Vis. 2004, 60, 91–110.
[CrossRef]
Snavely, N.; Seitz, S.M.; Szeliski, R. Modeling the world from internet photo collections. Int. J. Comput. Vis.
2008, 80, 189–210. [CrossRef]
Mathews, A.J.; Jensen, J.L.R. Three-dimensional building modeling using structure from motion: improving
model results with telescopic pole aerial photography. In Proceedings of the 35th Applied Geography
Conference, Minneapolis, MN, USA, 10–12 October 2012; pp. 98–107.
Pollefeys, M.; Gool, L.V.; Vergauwen, M.; Verbiest, F.; Cornelis, K.; Tops, J. Visual modeling with a hand-heldcamera. Int. J. Comput. Vis. 2004, 59, 207–232. [CrossRef]
Leberl, F.; Irschara, A.; Pock, T.; Meixner, P.; Gruber, M.; Scholz, S.; Weichert, A. Point clouds: Lidar versus 3D
vision. Photogramm. Eng. Remote Sens. 2010, 76, 1123–1134. [CrossRef]
Turner, D.; Lucieer, A.; Watson, C. Development of an unmanned aerial vehicle (UAV) for hyper resolution
mapping based visible, multispectral, and thermal imagery. In Proceedings of the 34th International
Symposium of Remote Sensing Environment, Sydney, Australia, 10–15 April 2011.
Dey, A.; Mummet, L.; Sukthankar, R. Classification of plant structures from uncalibrated image sequences.
In Proceedings of the IEEE Workshop on Applications of Computer Vision, Breckenridge, CO, USA,
9–11 January 2012; pp. 9–11.
Dandois, J.P.; Ellis, E.C. Remote sensing of vegetation structure using computer vision. Remote Sens. 2010, 2,
1157–1176. [CrossRef]
Wilburn, B.; Joshi, N.; Vaish, V.; Talvala, E.; Antunez, E.; Barth, A.; Adams, A.; Levoy, M.; Horowitz, M. High
Performance Imaging Using Large Camera Arrays. Trans. Graph. ACM 2005, 24, 765–776. [CrossRef]
Popovic, V.; Afshari, H.; Schmid, A.; Leblebici, Y. Real-time implementation of gaussian image blending in a
spherical light field camera. In Proceedings of the IEEE International Conference on Industrial Technology,
Cape Town, South Africa, 25–28 February 2013.
Nguyen, T.T.; Slaughter, D.C.; Max, N.; Maloof, J.N.; Sinha, N. Structured Light-Based 3D Reconstruction
System for Plants. Sensors 2015, 15, 18587–18612. [CrossRef] [PubMed]
Barone, S.; Paoli, A.; Razionale, A.V. A Coded Structured Light System Based on Primary Color Stripe
Projection and Monochrome Imaging. Sensors 2013, 13, 13802–13819. [CrossRef] [PubMed]
Mathews, A.J.; Jensen, J.L.R. Visualizing and quantifying vineyard canopy lai using an unmanned aerial
vehicle (UAV) collected high density structure from motion point cloud. Remote Sens. 2013, 5, 2164–2183.
[CrossRef]
Turner, D.; Lucieer, A.; Watson, C. An automated technique for generating georectified mosaics fromultra-high resolution unmanned aerial vehicle (UAV) imagery, based on structure from motion (SfM) point
clouds. Remote Sens. 2012, 4, 1392–1410. [CrossRef]
Snavely, N. Scene Reconstruction and Visualization from Internet Photo Collections. Ph.D. Thesis,
University of Washington, Seattle, WA, USA, 2008.
Tomasi, C.; Kanade, T. Shape and motion from image streams under orthography: A factorization method.
Int. J. Comput. Vis. 1992, 9, 137–154. [CrossRef]
Moreno-Noguer, F.; Lepetit, V.; Fua, P. Accurate Non-Iterative O(n) Solution to the PnP Problem.
In Proceedings of the IEEE International Conference on Computer Vision, Rio de Janeiro, Brazil,
14–21 October 2007.
Triggs, B.; Mclauchlan, P.; Hartley, R.; Fitzgibbon, A. Bundle adjustment—A modern synthesis. In Vision
Algorithms: Theory and Practice; Springer Berlin Heidelberg: Berlin, Germany, 2000; Volume 1883, pp. 298–375.

Sensors 2016, 16, 874

54.
55.
56.
57.
58.

18 of 18

Koniaris, C.; Cosker, D.; Yang, X.S.; Mitchell, K. Survey of Texture Mapping Techniques for Representing
and Rendering Volumetric Mesostructure. J. Comput. Graph. Tech. 2014, 3, 18–60.
Paulus, S.; Behmann, J.; Mahlein, A.-K.; Plümer, L.; Kuhlmann, H. Low-Cost 3D Systems: Suitable Tools for
Plant Phenotyping. Sensors 2014, 14, 3001–3018. [CrossRef] [PubMed]
Paproki, A.; Sirault, X.; Berry, S.; Furbank, R.; Fripp, J. A novel mesh processing based technique for 3D plant
analysis. BMC Plant Biol. 2012, 12. [CrossRef] [PubMed]
Nielsen, M.; Andersen, H.J.; Slaughter, D.C.; Granum, E. Ground truth evaluation of computer vision based
3D reconstruction of synthesized and real plant images. Precis. Agric. 2007, 8, 49–62. [CrossRef]
Roberts, R.; Sinha, S.N.; Szeliski, R.; Steedly, D. Structure from motion for scenes with large duplicate
structures. In Proceedings of the Computer Vision and Patter Recognition, Providence, RI, USA, 20–25 June
2011; pp. 3137–3144.
© 2016 by the authors; licensee MDPI, Basel, Switzerland. This article is an open accessarticle distributed under the terms and conditions of the Creative Commons Attribution(CC-BY) license (http://creativecommons.org/licenses/by/4.0/).