SPECTRAL UNMIXING OF FLUORESCENCE FINGERPRINT IMAGERY FOR
VISUALIZATION OF CONSTITUENTS IN PIE PASTRY
Naoto Yokoya1 , Mito Kokawa2 , and Junichi Sugiyama2
Department of Advanced Interdisciplinary Studies, The University of Tokyo, Japan
National Food Research Institute, National Agriculture and Food Research Organization, Japan1

2

ABSTRACTIn this work, we present a new method that combines fluorescence fingerprint (FF) imaging and spectral unmixing tovisualize microstructures in food. The method is applied tovisualization of three constituents, gluten, starch, and butter,in two types of pie pastry. It is challenging to discriminate
between starch and butter because both of them can be represented by similar FFs of low intensities. Two optimization
approaches of FF unmixing that consider qualitative knowledge are presented and validated by comparison to the conventional staining method. Although starch and butter wererepresented by very similar FFs, a constrained-least-squares
method with abundance quantization successfully visualized
the distributions of constituents in pie pastry.
Index Terms— Fluorescence fingerprint imaging, spectral unmixing
1. INTRODUCTION
A fluorescence fingerprint (FF), which is also known as an
excitation-emission matrix (EEM), is an alternative representation of the fluorescence spectra with each matrix element
representing the fluorescence intensity of a given pair of excitation and emission wavelengths [1, 2]. Owing to its large
amount of data, FFs can be used to discriminate between samples that have similar fluorescence properties, such as wines
made in different regions [3]. Recently, FF imaging has been
proved to be helpful for visualization of internal structures offoods, which is important for food monitoring [4, 5]. Conventional analytical techniques, such as principle component
analysis (PCA) and spectral angle mapper (SAM), are usefulfor extracting information from FF images; however, there is
still much room for advanced analysis to take advantage ofthe high dimensionality of FFs.
Hyperspectral imaging is a non-destructive imaging
method that utilizes the intrinsic spectral properties of constituents, and is a common macroscopic analytical tool forfood quality and safety control [6]. Spectral unmixing isa blind source separation technique actively developed inremote sensing using hyperspectral imaging [7]. Spectral unmixing refers to any process that separates the pixel spectrum

into spectra of distinguishable spectral signatures, which arecalled endmembers, and abundance fractions that represent
the percentage of each endmember that is present in the pixel.
Many researchers have studied unmixing based on a linear
spectral mixture model (LSMM), which simplifies the actual
complex physical model and considers the observed spectrum
as a linear combination of spectral signatures of endmembers.
FF imaging is a hyperspectral imaging method in the fluorescence mode. Although several researchers worked on
applying unmixing to fluorescent imaging, they are limited tothe use of the fluorescence spectrum, not the FF [8, 9, 10, 11].
In this work, we present FF unmixing, which decomposes
FF imaging data into endmember FFs and their abundance images, to non-destructively visualize the distributions of gluten
protein, starch and butter in food. Proteins, starches and fat
make the structure of food, and a visualization method ofthese three constituents is expected to have wide spread applications. Pie pastry is used as samples, since the three constituents are known to be distributed in relatively large clusters. Endmember FFs are extracted by a joint use of manual
selection of the reference FFs and a data-driven approach thatfinds vertices of data simplex. We present two abundance estimation methods: one approach formulated as a constrainedleast-squares problem, one using a matrix factorization technique. The effectiveness of FF unmixing is validated by comparison to the conventional staining method [12, 13].
2. MATERIALS AND METHODS
We focused on two typical types of pie pastry, puff pastry andshort pastry. Although the constituents of our interests aregluten, starch, and butter, it was necessary to add the fluorescence patterns of ferulic acid and microscope slide that represents air bubbles in pastry dough, in order to obtain decent
results. Therefore, we aim to decompose the FF of each pixel
into the FFs and abundance fractions of five constituents, i.e.,
gluten, starch, butter, ferulic acid, and microscope slide.
2.1. Image Acquisition
Two types of pastry dough were cut into pieces approximately 1 cm3 , embedded in compound (3% CMC embeddingmedium, iTec Science, Ibaraki, Japan) and frozen immediately in the cooling bath of a cold trap (Eyela UT-2000,
Tokyo Rikakikai Co. Ltd, Tokyo, Japan) with hexane as thecooling medium. When the samples were completely frozen,
the samples were sliced to 10 µm using a cryotome (CM1900, Leica) with a Surgipath DH80HS blade (Leica). The
thin slices were mounted on a microscope slide (S-8215 andS-9901, Matsunami Glass Ind., Ltd., Osaka, Japan) and kept
at -20 ◦ C until observation
FF images of two types of pastry dough and fractionatedgluten were acquired with the FF imaging system, using theexcitation and emission wavelengths selected with PCA. The
FF images were acquired in excitation and emission wavelengths ranging from 270 to 330 nm and 350 nm to 420 nm,
respectively, at 10 nm intervals.
2.2. Linear Spectral Mixture Model
The LSMM is adopted for its mathematical simplicity andphysical interpretability. The LSMM assumes that an observed FF at a pixel can be expressed as a linear combinationof endmember FFs:(1)

y = Ea + n.

Here, y ∈ RL×1 is the observed FF reshaped into one dimension, E ∈ RL×M is the endmember FF matrix with eachcolumn vector representing an endmember FF when it is reshaped into a matrix, a ∈ RM ×1 is the abundance vector, andn ∈ RL×1 is the noise vector. L is the number of combinations of excitation and emission wavelengths and M is thenumber of endmembers. When the LSMM is written for awhole FF image, the equation is expressed in matrix form as(2)

Y = EA + N,

where Y ∈ R
is the whole image, A ∈ R
is theabundance matrix, N ∈ RL×P is the noise matrix, and P is
the number of pixels. The FF unmixing problem is to decompose Y into E and A. When the number of endmembers are
known, spectral unmixing is mainly composed of two steps:
endmember extraction and abundance estimation.
L×P

M ×P

2.3. Endmember Extraction
A condition that assumes the existence of pure pixels for all
endmembers is called as pure-pixel assumption. In an area
corresponding to a pure pixel, there is only one endmember.
With this condition, the endmember spectra can be assumed
to locate on the vertices of the data simplex. In this case,the estimation of endmember spectra is boiled down as thegeometry-based detection of the vertices of the data simplex
in high-dimensional space. Endmember extraction methods
based on the pure-pixel assumption have been actively developed in remote sensing [14, 15, 16]. The vertex component analysis (VCA) algorithm is one of the most well-knowgeometry-based endmember extraction methods. VCA iteratively projects data samples onto a direction that is orthogonal to the subspace defined by the previously determined endmembers. The extreme point on the projection is detected as
the new endmember spectrum.
Unsupervised endmember extraction methods enablefinding vertices of the data simplex; however, a reference
spectral library is required to determine their physical properties. We jointly use the reference FFs and VCA to extractthe endmember FFs. First, we prepare the reference FFs.
The reference FF of gluten is obtained from the FF imageof fractionated gluten. Small areas of starch and microscopeslide were manually extracted from each observed FF image.
Those of butter and ferulic acid were extracted from the FF
image of the puff pastry dough owing to their clear textures.
All the FFs of the selected areas were averaged to acquire thereference FFs of these five endmembers. Since the variancesof FFs of gluten and ferulic acid are large, their endmember FFs were chosen from similar FFs extracted by VCA.
The cosine similarity is used for similarity measurement.
The endmember matrix can be derived using this supervised
endmember extraction.
2.4. Constrained-Least-Squares Method with Abundance
Quantization
Once the endmember signatures are obtained, the abundances can be estimated by minimizing the residuals in the
LSMM. Therefore, the unmixing problem is formulated as
the constrained-least-squares problem [17]:minimize 12 ∥y − Ea∥22!M
subject to a ≽ 0, i=1 ai ≤ 1,

(3)

where the vector a is the optimization variable. The inequality notion for vectors denotes componentwise inequality. The
first constraint is the nonnegativity of abundance fractions andthe second constraint means that the sum of abundance fractions at the pixel is less than unit due to variability of endmember spectra. This optimization is a quadratic programming problem [18], which is written as
minimize 12 aT E T Ea − y T Ea
subject to Ga ≼ h,(4)T

where G = [−I 1M ] (1M ∈ RM ×1 ), I ∈ RM ×M is the"#Tidentity matrix, and h = 0TM 1 (0M ∈ RM ×1 ).
In this work, we introduce quantization of the abundance
fraction of microscope slide. When the samples were sliced,
small holes were observed in the thin slice, where the microscope slide could be seen through. In this case, the abundance
fraction of microscope slide can be assumed to be 0 or 1. The
penalty term is added to the cost function in (3) as
1
1
2∥y − Ea∥22 + λ (aslide − U (aslide )) ,
2
2

(5)

(7)

minimize 12 aT (E T E + λC)a − (y T E + λbT )asubject to G2 a ≼ h2 ,(8)otherwise

where b ∈ R
has unit for bslide and zero values for the"#Tother components and C = bbT . G1 = GT b , h1 =
" T
#T
" T
#T
" T
#T
h 1/2 , G2 = G − b and h2 = h − 1/2 . In
our implementation, the optimization (4) is firstly solved. Sequentially, depending on the condition of aslide , the optimization (7) or (8) is solved to obtain the final result. Optimization
methods for quadratic programming, such as active-set andinterior-point methods, can solve (4), (7), and (8) with theglobal optimum. We refer to this method as the constrainedleast-squares method with abundance quantization (CLSAQ).
2.5. L1/2 Nonnegative Matrix Factorization
When we consider the nonnegativity of the matrices Y , E,
and A in (2), the unmixing problem can be seen as nonnegative matrix factorization (NMF), which factorizes a nonnegative matrix into two nonnegative matrices [19]. In the last
decade, NMF and its extensions have been intensively studied in hyperspectral unmixing and proved their effectivenesswithout the pure-pixel assumption [20, 21]. NMF was also
applied to fluorescent imaging data [10]. For our problem
setting, very few endmembers can be assumed to be mixed
in a pixel owing to high-spatial-resolution imaging and thelimited number of endmembers. Therefore, as a comparison
method to CLSAQ, we adopt a sparsity constrained NMF thatuses L1/2 sparsity constraint on abundances [22]. L1/2 -NMF
formulates the unmixing problem as(9)with variables E and A, where ∥ · ∥F denotes the Frobenius
norm, ∥A∥1/2 is the summation of square roots of all elements
in A and η is the parameter. This optimization is guaranteed
to converge to local optima using the multiplicative update
rules as follows:E ← E. ∗ Y% AT ./EAAT&
1
A ← A. ∗ E T Y./ E T EA + η2 A− 2 ,

380

1000

370
500
360
350
270 280 290 300 310 320 330

0

400

150

390
380

100

370
50

360
350
270 280 290 300 310 320 330

Freulic acid

420
700

410

600

400

500

390

400
380

300

370

200

360

100

350
270 280 290 300 310 320 330

0Excitation wavelength (nm)Butter420

0

Excitation wavelength (nm)

Excitation wavelength (nm)

420

200

Emission wavelength (nm)

1500

390

Emission wavelength (nm)

400

250

410

250

410
200

400

150

390
380

100

370
50

360
350
270 280 290 300 310 320 330

0Excitation wavelength (nm)Slide glass
250

410
200

400

150

390
380

100

370
50

360
350
270 280 290 300 310 320 330

0Excitation wavelength (nm)Fig. 1. Endmember FFs of short pastry dough.

M ×1

minimize 12 ∥Y − EA∥2F + η∥A∥1/2!M
subject to E ≽ 0, A ≽ 0,
i=1 ai ≤ 1,

Starch420
2000

Emission wavelength (nm)minimize 12 aT (E T E + λC)a − y T Ea
subject to G1 a ≼ h1 ,Gluten410

Emission wavelength (nm)This cost function is no longer convex. Depending on theabundance fraction of microscope slide, the optimization canbe divided into the two cases of quadratic programming: if
aslide ≤ 12 ,420

Emission wavelength (nm)where aslide is the abundance fraction of microscope slideand λ is a parameter. U (x) is a step function defined as$
0 if x ≤ 12
U (x) =
(6)
1 otherwise.

(10)

where .* and ./ denote elementwise multiplication and division, respectively. E and A are initialized using the extracted
endmember FFs and a matrix with constant values, respectively. To satisfy the additivity constraint, we add the black
spectrum to E, and the data matrix Y and the endmember
FF matrix E are augmented by a row of constants defined by
"
#T
"
#T
Ỹ = Y T δ1P and Ẽ = E T δ1P , respectively, whereδ controls the impact of the additivity constraint [17].
3. EXPERIMENTAL RESULTS
Fig. 1 shows the endmember FFs used for the short pastry
dough. Gluten and ferulic acid have characteristic fluorescence patterns, whereas those of starch and butter are very
similar. Although starch itself is not fluorescent, starch granules are known to be surrounded by a protein membrane. The
FF of starch is a mixture of the protein FF and the FF of
the microscopy slide which passes through the starch granule. The microscope slides are not completely fluorescence
free and show weak fluorescence.
Fig. 2 shows the abundance images of gluten, starch,butter, ferulic acid, and microscope slide obtained by CLSAQ
and L1/2 -NMF. The top two rows are the results of the short
pastry dough and the bottom two rows are those of the puff
pastry dough, which shows a band of butter sandwiched between two layers of wheat dough. As shown in the first
and third rows obtained by CLSAQ, abundances for microscope slide are shown in black (abundance=0) and white(abundance=1), whereas the other constituents are shown incontinuum values between 0 and 1. Even though the FFs ofstarch and butter are similar, there are obvious differences between their distributions for both pastry dough. Particularly,
in the abundance image of starch for the puff pastry dough,starch particles are visualized. As shown in the second andfourth rows obtained by L1/2 -NMF, the abundance images ofgluten and ferulic acid are very similar to those of CLSAQ;
however, it is difficult to distinguish those of starch and butter
for the short pastry dough and starch particles are failed to bevisualized for the puff pastry dough. These results indicate

Ferulic acid

Slide glass

L1/2 -NMF

CLSAQ

Butter

CLSAQ

Puff pastry doughStarchL1/2 -NMF

Short pastry dough

GlutenFig. 2. Abundance images of gluten, starch, butter, ferulic acid, and microscope slide from left to right columns for short pastry
dough on top two rows and puff pastry dough on bottom two rows.

Fig. 3. (Left) Composite image of the abundance imagesof butter (red), starch (green) and gluten (blue) obtained by
CLSAQ. (Right) Stained image of the same sample. Protein
is stained blue and fat is stained orange.
that it is important to place high priority on the reference FFs
carefully selected using human knowledge when endmember
FFs are very similar.
In order to validate the analysis results, the short pastry
was stained. Fig. 3 shows the composite image obtained by
CLSAQ assigning the abundance images of butter, gluten, andstarch to red, blue, and green, respectively, and the stained
image. Large aggregates of gluten can be observed in both
images and the positions of starch granules and fat in theanalyzed image are largely correspondent with those of thestained image. The FFs of starch and butter were difficult
to distinguish by visual judgmental; however, it was possible to obtain their abundances accurately. Fluorescenceintensity would be proportional to the volume of these constituents. In the short pastry, the volume ratio calculated
from the weights and specific gravities of the ingredientsis 12.9%:37.8%:49.3% for gluten, starch, and butter ratio.
On the other hand, the ratios calculated from the abundance matrices obtained by CLSAQ and L1/2 -NMF were
16.6%:37.6%:45.8% and 30.8%:36.3%:32.9%, respectively.
The ratio obtained by CLSAQ is very close to that calculated
from the ingredients of pie pasty, and seems to support theaccuracy of CLSAQ.
4. CONCLUSION
We have presented spectral unmixing of FF images to visualize distributions of the constituents in pie pastry. The experimental results showed that the constrained-least-squares
method with abundance quantization successfully visualize the abundance distributions of five endmembers eventhough there are very similar FFs. FF unmixing enables
non-destructive visualization of microstructures in biologicalimaging, and can be expected to be an alternative tool ofstaining.
5. ACKNOWLEDGEMENT
The authors thanks Ms. Hiroko Ashida (Fuji Oil Co., Japan)
for providing the experimental data.

6. REFERENCES[1] D. W. Johnson, J. B. Callis, and G. D. Christian, “Rapid
scanning fluorescence spectroscopy,” Anal. Chem., vol.
49, no. 8, pp. 747A–757A, 1977.

[11] F. Fereidouni, A. N. Bader, and H. C. Gerritsen, “Spectral phasor analysis allows rapid and reliable unmixingof fluorescence microscopy spectral images,” Optics Express, vol. 20, no. 12, pp. 12729–12741, 2012.

[2] B. J. H. Matthews, A. C. Jones, N. K. Theodorou, and A.
W. Tudhope, “Excitation-emission-matrix fluorescence
spectroscopy applied to humic acid bands in coral reefs,”
Marine Chemistry, vol. 55, pp. 317–332, 1996.

[12] F. Auger, M. H. Morel, J. Lefebvre, M. Dewilde, and A.
Redl, “A parametric and microstructural study of the formation of gluten network in mixed flour-water batter,”
Journal of Cereal Science, vol. 48, no. 2, pp. 349–358,
2008.

[3] C. Yin, H. Li, C. Ding, and H. Wang, “Preliminary investigation on variety, brewery and vintage of wines using three-dimensional fluorescence spectroscopy,” Food
Science and Technology Research, vol. 15, no. 1, pp.
27–38, 2009.

[13] T. Maeda, M. Kokawa, M. Miura, T. Araki, M. Yamada,K. Takeya, and Y. Sagara, “Development of a novel
staining procedure for visualizing the gluten–starch matrix in bread dough and cereal products,” Cereal Chem.,
vol. 90, no. 3, pp. 175–180, 2013.

[4] M. Tsuta, K. Miyashita, T. Suzuki, S. Nakauchi, Y.
Sagara, and J. Sugiyama, “Three-dimensional visualization of internal structural changes in soybean seeds
during germination by excitation-emission matrix imaging,” Transactions of American Society of Agricultural
and Biological Engineers, vol. 50, no. 6, pp. 2127–2136,
2007.

[14] J. W. Boardman, “Automating spectral unmixing of
AVIRIS data using convex geometry concepts,” Proc.
Ann. JPL Airborne Geosci. Workshop, vol. 1, pp. 11–
14, 1993.

[5] M. Kokawa, K. Fujita, J. Sugiyama, M. Tsuta, M. Shibata, T. Araki, and H. Nabetani, “Quantification of the
distributions of gluten, starch and air bubbles in doughat different mixing stages by fluorescence fingerprint
imaging,” Journal of Cereal Science, vol. 55, pp. 15–21,
2012.

[15] M. E. Winter, “N-findr: An algorithm for fast autonomous spectral end-member determination in hyperspectral data,” Proc. SPIE, vol. 3753, pp. 266–275,
1999.
[16] J. M. P. Nascimento and J. M. B. Dias, “Vertex component analysis: A fast algorithm to unmix hyperspectral
data,” IEEE Trans. Geosci. Remote Sens., vol. 43, no. 4,
pp. 898–910, 2005.

[6] A. A. Gowen, C. P. O’Donnell, P. J. Cullen, G. Downey,
and J. M. Frias, “Hyperspectral imaging–an emerging
process analytical tool for food quality and safety control,” Trends in Food Sciend and Technology, vol. 18, pp.
590–598, 2007.

[17] D. C. Heinz and C.-I. Chang, “Fully constrained leastsquares linear spectral mixture analysis method for material quantification in hyperspectral imagery,” IEEE
Trans. Geosci. Remote Sens., vol. 39, no. 3, pp. 529–
545, Mar. 2001.

[7] J. M. Bioucas-Dias, A. Plaza, N. Dobigeon, M. Parente, Q. Du, P. Gader, and J. Chanussot, “Hyperspectral
unmixing overview: geometrical, statistical, and sparse
regression-based approaches,” IEEE J. Sel. Topics Appl.
Earth Observ. Remote Sens., vol. 5, no. 2, pp. 354–379,
2012.

[18] S. Boyd and L. Vandenberghe, Convex Optimization.
Cambridge, U.K.: Cambridge Univ. Press, 2004.
[19] D. D. Lee and H. S. Seung, “Learning the parts of objects by nonnegative matrix factorization,” Nature, vol.
401, pp. 788–791, 1999.

[8] T. Zimmermann, “Spectral imaging and linear unmixing
in light microscopy,” Adv. Biochem. Engin./Biotechnol.
vol. 95, pp. 245–265, 2005.

[20] L. Miao and H. Qi, “Endmember extraction from highly
mixed data using minimum volume constrained nonnegative matrix factorization,” IEEE Trans. Geosci. Remote
Sens., vol. 45, no. 3, pp. 765-777, 2007.

[9] B. Kraus, M. Ziegler, and H. Wolff, “Linear fluorescence unmixing in cell biological research,” Modern Research and Educational Topics in Microscopy (ed. by A.
Mendez-Vilas and J. Diaz), pp. 863–872, 2007.

[21] S. Jia and Y. Qian, “Constrained nonnegative matrix
factorization for hyperspectral unmixing,” IEEE Trans.
Geosci. Remote Sens., vol. 47, no. 1, pp. 161-173, 2009.

[10] R. A. Neher, M. Mitkovski, F. Kirchhoff, E. Neher, F. J.
Theis, and A. Zeug, “Blind source separation techniques
for the decomposition of multiply labeled fluorescence
images,” Biophysical Journal, vol. 96, pp. 3791–3800,
2009.

[22] Y. Qian, S. Jia, J. Zhou, and A. Robles-Kelly, “Hyperspectral unmixing via L1/2 sparsity-constrained nonnegative matrix factorization, ” IEEE Trans. Geosci. Remote Sens., vol. 49, no. 11, pp. 4282–4297, 2011.