ple of operation, prototype
designed for scientific multidimensional images
Rakutenal processing analysis,
asInstitute of Technology
http://rit.rakuten.co.jp
Figure 3 is an exemplary image capture of a spectrum from
cations and evaluationof
楽天技術研究所
rakutenRIT
cotton towel. From end to end, the spectrum is 160 pixel
ons, and share our plan for
and we measured the intensity of each frequency of lighusing plot profile function. The function converts the pointon the spectrum indicated as white line to gray scale anthere is no previous work
lists the intensity of each point as value of 0 to 255.
of spectrophotometry fortion purposes. However, aercial products have beenions of the issues we haveSlitmputer VisionEnhancing(CV) based E-Commerce Product Description by Leveraging
to Represent Garment Color and Airinessrom parts Spectrophotometryof an image
diffraction
g the popular[1,2,3].
Our (qq146419@iii.u-tokyo.ac.jp) & Adiyan Mujibiya (adiyan@acm.org)
Shogo
Yamashita
grating
motivation in providing
hermore seeks to issue:
extendrepresenting material texture in e-commercemination by incorporatingUSB
camerareflected light’s  These testedandgarments
fixed
the spectrometerand theeach fmThesebut tested
also garments
near-infrared
are fixed
thespectrum.
spectrometerthearesurfaceofeach fabricis surface
fixedof toaletteP

140

Light Intensity [0-255]

Light Intensity [0-255]

at programa same position
on top of thecase, whichwasmade using
do not affectto the s
descrip-on	
  oﬀers	  very	
  limited	
  informa-on	
  
of	
  the
mdistance
aterial	
  difference
texture	  [7], which is an open source image-processingProduct	
  fdesigned
the case,which
was madeusing
the distance difference
the onspectrum.
Figurea 3D printer dowith notblack affectplastic filament
8 istoa result
the experiment
for recogni
for scientificmultidimensional
imagesColor	
  informa-on	
  is	
  also	  not	  so	
  reliable	
  due	  to	
  ofdiﬀerent	  mbient	  igh-ng	  fabric.
a) , ab),c) andd)lairinessis spectrum o
2:
isforaexperiment
systemthe
based
on parts
activespectrometer,stic
is aonresulton theforform
recognizing
theFigurefilament
3 is an exemplary image capture of a spectrum
Figurefrom
6 isFigure
a8result
thePalette
experiment
recognizingofMicrofiber
Towel,
This	
  
o
<en	
  
h
inders	  costumer	
  
f
rom	
  
b
uying	  products	
  
o
nline
ics
cotton of
towel. color
From end to description
end, the spectrum is color
160 pixels
on fabric.
a) is original
spectrumofdiffraction
thewhited)LEDoffabric.
a)
,
b),
c)
and
is
spectrumonreflected
light
which
consistsofagratingandaconventional
HandkerchiefandT-Shirt, respectively.
andwemeasuredtheintensityofeachfrequencyoflightmeasured by directly emitting the light to the spectrometer.
erchexperimentfor
recognizingthedifference seems
to be causedby differen
been
proposedusinghasplot profile
function.
The function [4].
convertsb),thec)points
Microfiber
Towel,
CottonTowel,
USBBluecameraand d)form
is spectrumpartsof reflectedof
light form
White,
thicknessofthefabric.
Inaddition,
there aon thespectrum
spectrum indicatedwhiteline to grayscaleGreenand fabric, respectively. White fabric refracts all
inalof asthe
white
LED
and
sible,NaturalHandkerchiefandT-Shirt,respectively.
Theon each spectrum,
whichamplitude
probably can be ulists the intensityof each pointLanguage
as value of 0 to 255.
range frequency oflight. Therefore,
the spectrum
indicatesting thelight to thespectrometer.
proposed
systemarchitecture:quantify
color
andtextureusing
spectrometermaterial.
reflectedcharacteristic
light’s  spectrum.
testedLED’s  garmentsareand the surfaceofof eachfabric is fixed
difference
seems
tospectrum.
befixedcausedbythe
difference
airinessor tosimilarto These
original
On thethespectrometer
limitations
when
exposedatasame
position
on
top
of
the
case,
which
was
made
using
do
not
affectthe
distancedifferencetothespectrum.
Figuref reflected light form White, Blueother hand, the characteristic of spectrum on blue and green
a 3D printer withblack plastic filament
8 is a result onthere
the experiment
for recognizing
the airinessthicknessofthe one.
fabric.
In addition,
are somefeatures.tively.
Realizing
thisissue,
anfabricisdifferent
fromtheoriginalBluefabriconlyWhite fabric refractsSlit allof fabric. a) , b), c) and d) is spectrum on reflected light
Figure 6specificis aonresulton the experiment
recognizing
theprobably
reflects
frequency
for blue for(380
to495nm).
Green
each
spectrum,whichused Towel,
to recognize
form partscanof beMicrofiber
Cotton Towel,edonMechanicalTurkforcolor on fabric. a) is original spectrum of the white LED
Therefore, the spectrum indicatesfabricreflects frequency for blue and green
(455
to
630
nm).
HandkerchiefandT-Shirt,respectively.
amplitude
Diffracted difference seems to be caused by differenceTheof airinessmeasured by the
directly
emitting the light to the spectrometer.
material.
ordiffraction This information can be used to recognize accurate color on
conducted
[6].
This
work
b),
c)
and
d)
is
spectrumofreflected
light
form
White,
Blue
riginal LED’s   spectrum.grating
On theSlitthickness of the fabric. In addition, there are some features
lightfabric.
and Green fabric, respectively. White fabric refracts allon each spectrum,
which probably Cotton
can be used to recognize
Microfiber
ticof spectrum
on blue and green
e metrics
for texture.
range frequency of light. Therefore, the spectrum indicatesdiffractionlens CCD
Handkerchie
the material.
similarcharacteristic
tooriginal
LED’s  spectrum.
On
the
Towel
Towel
USB
White
USB
he original one. Bluefabriconly
gratingsensorother hand, the characteristic of spectrumLighton blue and green
cameraFigure 7: Materials used for texture r
fabric is different
from the original one.Source
Blue fabric only camera
such
as (380surface
texturelightfor blueto 495nm).
Greenreflects specific frequency for blue (380 to 495nm). Green
Figure 2: Palette is a system based on active spectrometer,fabric reflects frequency for blue and green (455 to 630 nm).
s alsobeen
proposed[5].
which consistsofa diffraction
gratinga conventional
or
blue
and
green
(455
to and
630
nm).
This
informationcanbe usedaccurate color
on
Figure3:to recognize
Diffraction
gratingdiffracts incident light andBlue
USB camera
White
Green
Spectrometer
fabric.
buted
strainand oned to recognize
accurate
color
result:gauges
representing
garment color andtextureusing
spectrophotometry
FabricCottonlens makethese Microfiberparallellights.
T -Shirt
120
100

140
120
100

F) films embedded in Figure 5: Materials used for color
recognition and theLighttest setting
color
detection
texturedetection
Figure 7: M
aterials used
for texture recognition
experimentenvironment
MicrofiberSource
CottonDiffracted
Different
texturesinduceT -Shirt
Handkerchief
Slit
light
Towel
Towel
diffraction lens CCD
ions,
and
White Green Blue
Spectrometer
White
USB
gratingLightconsequently,
sensorFabriclight
camera7: Materials
used for texture recognition
the presenceof different Figure 5: Materials usedFigure
Sourceforcolor
recognition
andtheFigure 3: Diffraction grating diffracts incident light andexperiment environment
lens make theseparallel lights.
unately, bulky
settings
andsynchronization of scan
140140a) White LEDb) White Fabric
160 pixels
Spectrometer120
120
l   poses  challenges   in   end100
100
Fabric80
80
pproaches that incorporate
Figure 4: Exemplary image captureof a spectrum fromtested materials 160 pixels
60
60
for
color
recognition
andthecotton towel.
s toFigure
explorefeatures40
40Figure 8: Spectrums of reflected light from
4: Exemplarynew
image captureof a spectrum fromnt environment cotton towel.
2020 c) and d) is spectrum on reflected light f
and texture more robustly.
0
0 Microfiber Towel, Cotton Towel, Handke
Towel

Microfiber
CottonSpectrometer
Towel
Towel

T -Shirt

Handkerchief

Fabric

80
60

490

590

690

140
120

490

80

WaveLength [nm]

590

690

60
40

c) Blue Fabric

140
120

100

WaveLength [nm]

390

80

490

590

690

WaveLength [nm]40Light Intensity [0-255]

20

690

390

490

Wa

140
120

40
20

390

490

140
120

b) Cot

80
60
40
20

0

100

590

690

80

390

490

590

Light Intensity [0-255]

60

WaveLenglth [nm]a) Microfiber 40
Towelb) Cotton Towel

60
20

0

140

390

120

490

80
60

100
80
60
40
20

0

390

20

0

490

590

690

WaveLenglth [nm]c) Handkerchief

763.65,
1

490

W

80
60

c) Handkerchief4040d) Green Fabric

120

WaveLenglth100
[nm]

100

390

140

590 120 690

763.65,
1

140

690

WaveLenglth [nm]

60

d)

20

0

390

490

590

690

763.65,
1

WaveLenglth [nm]d) T-Shirt

490
590
690
390
490
590 Shirt,
690respectively0reflected lightfrom fabrics. a) , b),
Figure3906: Spectrums
fromcolored490
590
690 of reflected
390 light
490
590
690
WaveLenglth[nm] fabric. Figure 8: Spectrums ofWaveLenglth[nm]
100
Materials absorb
specific
frequency (or wavelength)of WaveLength
experimentenvironment
WaveLength [nm][nm]c) andEVALUATIONSd) is spectrum on reflected light form parts ofa) is original
spectrumof the whiteLED. b), c) and d) is
Spectrometer
Microfiber Towel, Cotton Towel, Handkerchief and T80
Fabriclights dependingon the color of the material. Therefore,c) Blue
Fabricd) Green
FabricBlue and Green
spectrumofreflected
light
form
White,
To evaluate
therespectively
effectiveness of our approa
color recognition and thea)light
Microfiber Towelb)Shirt,
CottonTowel
reflected 60light from material has information on color.
Ournvironment
respectively
Figure6: Spectrums
offabric,
reflecteda) Microfiber
Towelb) Cotton
Towel from colored fabric.
the thickness and airiness of each garment.
spectrophotometry
framework is able to measurea) each
EVALUATIONS
is original spectrum of the white LED. b), c) and d) is40
correlates with the spectrums mean amp-140form
New
YorkBlue-andParis
TextureRecognition
spectrumofTokyoreflected light
White,
Green
wavelength’s light intensity. Therefore, true colorof theTo evaluate140
the effectiveness of our approach, we measure
40

390

40
20

0

390

20
490

Light Intensity [0-255]

Light Intensity [0-255]

60

100
80
60

690

40
20

0

120
80
40

590

690

20

390

WaveLenglth [nm]

490

590

390

490

690

WaveLenglth [nm]b) Cotton Towel

590

690

100
80
60

Color Recognition
40
20

Materials
specific frequency (or wavelength) o
globalabsorb
one teamlights depending on the color ofrespective
the material.
Therefore
garment. Figure 9 depicts our ex
490

590

690

390

WaveLenglth [nm]

60

20

0

0

390

100

a) Microfiber
Towel
0

590

60

0

490

40

20

140
120
100

490

590

690

WaveLenglth [nm]

140
120
100

]

60

80

Light Intensity [0-255]

80

Light Intensity [0-255]

tensity [0-255]

100

140
120
100
80
60
40
20
0

100

80

]

140

components, which are
opy and CCD sensor for
120

100

y [0-255]

Intensity [0-255]

140

y [0-255]

Light

Figure 7: Materials used for texture recognition
Source Recognition
Color
Figure 5: Materials used for color recognition and the

39020

590

Light Intensity [0-255]

100

Cotton
T40-Shirt
Handkerchief
Towel20a) White LEDb)20White FabricFigure0 7: Materials used for texture recognition
010039080Light Intensity [0-255]

690

Light Intensity [0-255]

390

40

100
a) Microfiber Towel

100

Light Intensity [0-255]

Light Intensity [0-255]

590

40

Light Intensity [0-255]

690

60

WaveLenglth [nm]

0

60

0

80

0

490

120

80

20

20

140

WaveLength [nm]

100

40

390

Light Intensity [0-255]

Light Intensity [0-255]

Microfiber
40
Towel

590

WaveLenglth [nm]Light Intensity [0-255]

White Green Blue

490

60

Light Intensity [0-255]

Light
Source

140
120
100
80
60
40
100
20
80
0
390

Light Intensity [0-255]

the material.

Light Intensity [0-255]
Light Intensity [0-255]

refore, the spectrum indicates
nal LED’s   spectrum. On theof spectrum on blue and green
riginal one. Blue fabric only
blue (380 to 495nm). Greenlue and green (455 to 630 nm).
to recognize accurate color on

WaveLenglth [nm]Light Intensity [0-255]

Figure 6 is a result on the experiment for recognizing thecolor on fabric. a) is original spectrum of the white LED
measured by directly emitting the light to the spectrometer.
b), c) and d) is spectrum of reflected light form White, Blueand Green fabric, respectively. White fabric refracts all
range frequency of light. Therefore, the spectrum indicatessimilarcharacteristic
totheis fixed toese tested
garments
are fixed
the original
spectrometerLED’s  and the spectrum.
surface of eachOnfabrice case,whichhand,
was made
do not affectthe distance difference
the green
spectrum. Figure
othertheusing
characteristicof spectrum
on blue toand
filament8
isaresultontheexperimentfor
recognizing
the airinessfabric is different from the original one. Blue fabric onlyof fabric. a) , b), c) and d) is spectrum on reflected light
xperiment
for recognizing
reflects
specific thefrequency
blue
to 495nm).
form forpartsof (380
Microfiber
Towel, Green
Cotton Towel,l spectrum of the white LED
Handkerchief
and green
T-Shirt, (455respectively.
reflects
frequency
for blue andto 630The
nm).amplitude
the fabriclight to thespectrometer.
difference seems to be caused by difference of airiness orThis
can beused to recognize accurate color onflectedlightinformation
form White, Blue
thickness of the fabric. In addition, there are some features
ly. Whitefabric refracts all
fabric.
on each spectrum, which probably can be used to recognize

Towel
60
0

Light Intensity [0-255]

the spectrometer and the surface of each fabric is fixed to
do not affect the distance difference to the spectrum. Figure
8 is a result on the experiment for recognizing the airiness100
of fabric.140a) , b), c) and d) is spectrum on reflected
light
120
form partsof Microfiber Towel, Cotton80 Towel,
100
60
80
Handkerchief
and T-Shirt, respectively. The amplitude40
difference60seems to be caused by difference of airinessor
40
thickness 20of the fabric. In addition, there are some20features0on each spectrum,which
probably
can
be
used
to
recognize
0
390
490
490
590
690
the material.390

Light Intensity [0-255]

reflected light’s   spectrum. These tested garments are fixed
at a same position on top of the case, which was made using
a 3D printer with black plastic filament

Handkerchief80