17th International Conference on Machine Vision Applications (MVA)
Fully Online, July 25–27, 2021.

P2-13

Illumination Planningfor Measuring Per-Pixel Surface Roughness
Kota Arieda
Takahiro Okabe
Kyushu Institute of Technology
okabe@ai.kyutech.ac.jp
Abstract
Measuring per-pixel surface roughness is useful formachine vision applications such as visual inspection.
The surface roughness can be recovered from specularreflection components, but a large number of images
taken under diﬀerent lighting and/or viewing directions
is required in general so that suﬃcient specular reflection components are observed at each pixel. In thispaper, we propose a robust and eﬃcient method for perpixel estimation of surface roughness. Specifically, we
propose an illumination planning based on noise propagation analysis; it achieves the surface roughness estimation from a small number of images taken under the
optimal set of light sources. Through the experiments
using both synthetic and real images, we experimentallyshow the eﬀectiveness of our proposed method and our
setup with a programmable illumination and a polarization camera.

1 Introduction
The appearance of an object surface is described by
the reflectance properties, i.e. BRDFs for opaque surfaces and BSSRDFs for translucent surfaces. The reflectance properties depends on surface materials suchas metals, plastics, and silicons and their surface statessuch as rust, cracks, and scratches. Therefore, estimating the reflectance properties of object surfaces isuseful not only for photorealistic image synthesis but
also for visual inspection of object surfaces such as
metallic surfaces [18, 11, 10, 9, 14] and printed circuitboards [12, 7]. In this study, we propose an imagebased method for measuring the reflectance propertiesof object surfaces, in particular per-pixel surface roughness in a non-contact and non-destructive manner.
In general, specular reflection reflects the light incoming from a light source to a narrow range of directions, while diﬀuse reflection uniformly reflects thelight, e.g. the radiance of a Lambertian surface is independent of viewpoints. For a smooth surface, specular reflection reflects the light to the mirror-reflection
direction, i.e. the outgoing angle is the same as theincoming angle, and then we observe sharp specular
highlights. As the roughness of an object surface increases, the specular reflection reflects the light to abroader range of directions, and then we observe moreblurred specular highlights. Therefore, we can estimate© 2021 MVA Organization

the surface roughness on the basis of specular reflection
observed on an object surface.
In order to measure per-pixel surface roughness, weneed to observe specular reflection components at eachpixel under diﬀerent lighting and/or viewing directions
in general so that they are suﬃciently observed at eachpixel. Therefore, a large number of images is required
for measuring per-pixel surface roughness; the numberof required images increases as the surface roughness
decreases, because the specular highlights are sharperfor smoother surfaces. In addition, acquiring a large
number of images with diﬀerent light source and/or
camera positions by mechanically moving a light source
and/or a camera is time consuming.
Accordingly, in this paper, we propose a robust andeﬃcient method for measuring per-pixel surface roughness. The key idea of our proposed method is illumination planning on the basis of noise propagation analysis. Specifically, our method estimates per-pixel surface
roughness of the Torrance-Sparrow model [13] from theimages taken under the optimal set of light sources bytaking noises in pixel values into consideration. It is
contrast to the existing technique [3] for robust andeﬃcient photometric stereo; the estimation of surface
normal assuming the Lambert model [16]. In addition,
our setup makes use of an LCD as a programmable polarized illumination and a polarization camera for distinguishing a specular reflection component from diffuse and subsurface scattering components.
The main contribution of this paper is twofold. Firstof all, we propose an illumination planning for estimating per-pixel surface roughness. Specifically, we
study the noise propagation in surface roughness estimation, and derive how to select the optimal lightsources for robustly and eﬃciently estimating surface
roughness from a small number of images. Second, we
confirmed the eﬀectiveness of our method through theexperiments using our prototype setup. Our setup eﬃciently controls light source positions with an LCD androbustly extracts specular reflection components usinga polarization camera.

2 Related Work
Our proposed method is related to the recovery of
spatially-varying reflectance functions. Since the recovery of spatially-varying reflectance is often underconstrained, in particular when specular reflection does

not observed at a surface point, data-driven approaches
has been studied.
Alldrin et al. [1], Hui and Sankaranarayanan [5], andHui et al. [6] propose methods for estimating spatiallyvarying reflectance by using both the multiple images
with photometric stereo setup and a collocated camera and light source, and the prior knowledge with respect to spatially-varying BRDFs such as basis BRDFs
and their dictionary. Recently, deep learning based approach is studied [8, 17, 2, 4]. They achieve the recovery of spatially-varying reflectance even from a single
image.
In contrast to the above methods, our proposed
method is the direct measurement of surface roughness, and therefore need not collect the training data
and achieves per-pixel recovery independent of otherpixels. In addition, our formulation enables us to predict the accuracy (MSE) of the estimated parameters(see Section 3.2 for more details).

3 Proposed Method
Our proposed method assumes that the position of acamera, the positions of light sources, and the normalof a surface point are known in advance, and then estimates two parameters of the Torrance-Sparrow model,
i.e. surface roughness and specular reflectance per
pixel. In this Section, we show that the estimation of
surface roughness results in weighted least squares andthat the optimal light sources can be selected on thebasis of noise propagation analysis. Then, we describe
the illumination planning for robustly and eﬃciently
estimating surface roughness.

since ln(i − δ) ≃ (ln i − δ/i) when δ ≪ i.
Our proposed method estimates the specular reflectance and surface roughness from multiple images
taken under diﬀerent light sources. We denote the pixel
value, the angle between the surface normal and thehalf vector, and the noise under the l-th light source(l = 1, 2, 3, ..., L) by il , βl , and δl respectively. Then,the set of linear equations, i.e. the constraints imposedby the observed pixel values are represented as()− ln k= −W a + δ.
(4)
WB
1/s2
Here, W , B, a, and δ are given by



1
i1
0
i2

 1

, B =  .
W =
..


 .
.
.
0
iL
1


β12 /22
β2 /2 .. 
,.
βL2 /2(5)a = (ln[(n⊤ v)i1 ], ln[(n⊤ v)i2 ], · · · , ln[(n⊤ v)iL ])⊤ , andδ = (δ1 , δ2 , · · · , δL )⊤ .
Therefore, we set δ = 0 in eq.(4), and estimate thespecular reflectance and surface roughness by solving
the set of linear equations as()− ln k= −(W B)+ W a.
(6)
1/s2
Here, (W B)+ is the pseudo inverse matrix of (W B),
i.e. (W B)+ = [(W B)⊤ (W B)]−1 (W B)⊤ . Intuitively, eq.(4) and eq.(5) say that the l-th light source
is not useful when il ≃ 0 or ∃l′ , βl ≃ βl′ .

3.2 Selection of Light Sources
3.1 Estimation of Surface Roughness
According to the (simplified) Torrance-Sparrowmodel [13], the pixel value of specular reflection component i observed at a surface point is represented by
i=

k

β2

n⊤ v

e− 2ss ,

i=

k
e
n⊤ v

+ δ.

We take logarithm of eq.(2), and then obtain()() − ln ki 1 β 2 /2
= −i ln[(n⊤ v)i] + δ,
1/s2

Σ = E[(x − x̄)(x − x̄)⊤ ],(1)

where the specular reflectance k and the surface roughness s are unknowns to be estimated. We assume thatthe surface normal n, the viewing direction v, and theangle β between the surface normal and the half vector
are known.
We assume that the observed pixel value is contaminated by additive noise δ as
β2
− 2s
s

Let us denote the estimated and ground truth
parameters of the Torrance-Sparrow model, i.e.
(− ln k, 1/s2 )⊤ at a surface point by x and x̄ respectively. We study the variance-covariance matrix Σ ofthe parameters defined by(7)

where E[ ] stands for the expectation values.
Substituting eq.(4) into eq.(7), we can deriveΣ= E[(W B)+ δ{(W B)+ δ}⊤ ]=E[(W B)+ δδ ⊤ {(W B)+ }⊤ ]=σ 2 (W B)+ {(W B)+ }⊤=σ 2 (B ⊤ W 2 B)−⊤ ,

(8)

(2)

where we assume independent and identically distributed noises whose mean and variance are 0 andσ 2 respectively. Since the mean square error (MSE)
is proportional to the trace of the variance-covariancematrix, we obtain the MSE of the parameters as(3)

MSE ∝ Tr[(B ⊤ W 2 B)−⊤ ].

(9)

0.01
0.003

0.002

0.001

LCD

proposed medhod
random selection

polarization
camera0

light sources on LCD
object surface points
camera(a)5

Our proposed method iteratively selects the optimal
light source for reducing the MSE (see Section 3.3 formore details). Since we have P pixels in total, ourmethod selects the light source so that
Tr[(Bp⊤ Wp2 Bp )−⊤ ]67
8
9number of light sources

10

0.008
0.006
0.004
0.002

proposed medhod
random selection

0
5

(a)

(b)

Figure 1. Experimental environment: (a) our
setup and (b) its geometry.

P∑

RMSE of reflectance

RMSE of roughness

object

(10)

6
7
8
9
number of light sources

10

(b)

(c)

(d)

L=5

L=6

L=7

L=8

Figure 2. Experimental results using synthetic
images: uniform surface.

p=1

is minimized. Here, Bp and Wp stand for the matrices
in eq.(5) at the p-th pixel (p = 1, 2, 3, ..., P ).

3.3 Illumination Planning
We can see that the optimal set of light sources depend not only on the known geometry but also on theunknown parameters of the Torrance-Sparrow modelthrough the matrix Wp in eq.(10). Accordingly, we
propose two approaches: online and oﬄine illumination plannings.
The online illumination planning iteratively adds
light sources for reducing the MSE of the estimatedparameters, where the parameters for computing the
pixel values in eq.(10) are updated every time the lightsource is added. Specifically, when we select the (l+1)th light source, we compute the pixel values under the(l + 1)-th light source by using the specular reflectanceand surface roughness estimated from the l images.
The oﬄine illumination planning assumes that someestimates of the parameters are known and then computes the optimal set of light sources in advance. As
the surface roughness decreases, the specular highlights
become sharper and the larger number of light sources
distributed more densely is required. Therefore, the optimal set of light sources are computed from the lower
limit of the surface roughness of an object of interest.

4 Experiments
4.1 Experimental Environment
To confirm the eﬀectiveness of our proposed method,
we conducted experiments using both synthetic andreal images. In the experiments using real images, weused the setup consisting of an LCD and a polarization
camera as shown in Figure 1 (a). Specifically, we displayed white blocks on the LCD and used them as polarized light sources; the total number of light sources

is 210. The images of an object of interest were captured by using the polarization camera BFS-U3-51S5PC from FLIR and the specular reflection components
were extracted from those images [15]. We assumed
planar objects as is often the case with CV-based visual inspection [18, 11, 12, 7, 10, 9, 14]. The geometry among the light sources, the points on the object
surface, and the camera was calibrated by using MATLAB camera calibration toolbox in advance (see Figure 1 (b)). In the experiments using synthetic images,
we rendered the images with the same geometry, andadded zero-mean Gaussian noises to them whose standard deviation is 0.01 for the pixel values scaled from0 to 1.
We compared our proposed online illumination planning with the random selection of light sources. Specifically, both the methods empirically select initial lightsources from 210 light sources so that we can observe
specular reflection components under at least two lightsources at each pixel, because we have two unknowns
to be estimated. Then, our method iteratively adds
the image taken under the optimal light source thatminimizes the sum of traces in eq.(10) and updates thesurface roughness and specular reflectance. We findthe optimal light source from the 210 candidates in abrute force manner. The random selection iteratively
adds the image taken under a randomly selected lightsource and computes the surface roughness and specular reflectance. The number of trials for the random
selection was 50.

4.2 Results Using Synthetic Images
First, we tested a surface with uniform reflectanceproperties; s = 0.05 and k = 0.3. Figure 2 shows (a)the RMSEs of the estimated surface roughness and (b)the RMSEs of the estimated specular reflectance vs.
the number of light sources. The solid and dotted lines

0.025RMSE of roughness

RMSE of roughness

0.001
0.00080.00060.00040.0002

proposed medhod
random selection

0.05
surface roughness

0.06

7

(a)

8
9
10
11
number of light sources

0.18

12

(c)

(d)

(d)

L=8

L=9

0.2
0.22
surface roughness

0.24

proposed medhod
random selection
6

(a)

(b)

(c)

L=7

0.01
0.005
0

0
0.04

0.02
0.015

L=10

L=6

7
8
9
10
number of light sources

11

(b)

L=7

L=8

L=9

Figure 3. Experimental results using synthetic
images: nonuniform surface.

Figure 4. Experimental results using real images:
ground glass.

stand for the RMSEs of our method and the random selection respectively. The error bars stand for the rangeof ±1 sigma. We can see that the RMSEs of both our
method and the random selection decrease as the number of light sources increases. In particular, we can seethat the RMSE of our method is smaller than that of
the random selection. In other words, our illumination
planning achieves the same accuracy with a smaller
number of light sources than the random selection.
Figure 2 visualizes (c) the error map of the surface
roughness estimated by using our proposed method and(d) the trace map in eq.(10), when the number of lightsources increases from 5 to 8. Here, the larger is thebrighter. We can see that the traces are correlated withthe errors of the estimated surface roughness, and thelight sources are added so that they reduce the traces.
This shows the eﬀectiveness of selecting light sourceson the basis of the sum of traces.
Second, we tested a surface with non-uniform reflectance properties; the distribution of surface roughness is shown in Figure 3 (a). Figure 3 (b) shows the
RMSEs of the estimated surface roughness vs. thenumber of light sources. Similar to the above result for
the uniform surface, we can see that the RMSE of our
method is smaller than that of the random selection.
We obtained the similar results to the above as to theerror map of the estimated surface roughness and thetrace map as shown in Figure 3 (c)(d). Those results
support the eﬀectiveness of our method for nonuniform
surfaces.

Figure 4 (b) shows the RMSEs of the estimated surface
roughness vs. the number of light sources. The solidand dotted lines stand for the RMSEs of our method
and the random selection respectively. We can see that
the RMSE of our method is smaller than that of therandom selection.
Figure 4 visualizes (c) the error map of the surface
roughness estimated by using our proposed method,and (d) the trace map in eq.(10), when the number oflight sources increases from 6 to 9. We can see that thetraces are correlated with the errors of the estimatedsurface roughness, and the light sources are added sothat they reduce the traces. Those results show theeﬀectiveness of our proposed method for real images.

4.3 Results Using Real Images
In the experiments using real images, we tested aground glass with almost uniform reflectance properties. We considered the surface roughness estimatedfrom all of the 210 images as the ground truth. Figure 4(a) shows the distribution of the surface roughness.

5 Conclusion and Future WorkIn this paper, we proposed a method for estimating per-pixel surface roughness on the basis of specular reflection components. Specifically, we studied
the noise propagation in surface roughness estimation,and derived how to select the optimal light sources forrobustly and eﬃciently estimating surface roughness
from a small number of images. We confirmed the effectiveness of our method through the experiments using our prototype setup consisting of a programmablepolarized illumination and a polarization camera.
Currently, our method assumes planar surfaces as isoften the case with CV-based visual inspection. The
extension to curved surfaces is one of the future direction of our method.

Acknowledgment
This work was supported by JSPS KAKENHI Grant
Numbers JP20H00612 and JP17H01766.

References[1] N. Alldrin, T. Zickler, and D. Kriegman, “Photometric stereo with non-parametric and spatially-varying
reflectance”, In Proc. IEEE CVPR2008, pp.1–8, 2008.
[2] V. Deschaintre, M. Aittala, F. Durand, G. Drettakis,
and A. Bousseau, “Single-image SVBRDF capture witha rendering-aware deep network”, ACM ToG, 37(4),pp.1–15, 2018.
[3] O. Drbohlav and M. Chantler, “On optimal light configurations in photometric stereo”, In Proc. IEEE ICCV2005, pp.II-1707–1712, 2005.
[4] D. Gao, X. Li, Y. Dong, P. Peers, K. Xu, and X. Tong,“Deep inverse rendering for high-resolution SVBRDF
estimation from an arbitrary number of images”, ACM
ToG, 38(4), pp.1–15, 2019.
[5] Z. Hui and A. Sankaranarayanan, “A dictionary-based
approach for estimating shape and spatially-varying
reflectance”, In Proc. ICCP2015, pp.1–9, 2015.
[6] Z. Hui, K. Sunkavalli, J. Lee, S. Hadap, J. Wang,
and A. Sankaranarayanan, “Reflectance capture using univariate sampling of BRDFs”, In Proc. IEEE
ICCV2017, pp.5372–5380, 2017.
[7] A. Ibrahim, S. Tominaga, and T. Horiuchi, “Spectral
imaging method for material classification and inspection of printed circuit boards”, Optical Engineering,49(5), 057201, 2010.
[8] Z. Li, Z. Xu, R. Ramamoorthi, K. Sunkavalli, and
M. Chandraker, “Learning to reconstruct shape andspatially-varying reflectance from a single image”, ACM
ToG, 37(6), pp.1–11, 2018.
[9] C. Liu and J. Gu, “Discriminative illumination: perpixel classification of raw materials based on optimal

projections of spectral BRDF”, IEEE Trans. PAMI,
36(1), pp.86–98, 2014.
[10] C. Liu, G. Yang, and J. Gu, “Learning discriminative illumination and filters for raw material classification with optimal projections of bidirectional texture
functions”, In Proc. IEEE CVPR2013, pp.1430–1437,
2013.
[11] F. Pernkopf and P. O’Leary, “Image acquisition techniques for automatic visual inspection of metallic surfaces”, NDT & E International, 36, pp.609–617, 2003.
[12] S. Tominaga and S. Okamoto, “Reflectance-based material classification for printed circuit boards”, In Proc.
IEEE ICIAP2003, pp.238–244, 2003.
[13] K. Torrance and E. Sparrow, “Theory for oﬀ-specularreflection from roughened surfaces”, JOSA, 57(9), pp.
1105–1114, 1967.
[14] C. Wang and T. Okabe, “Joint optimization of codedillumination and grayscale conversion for one-shot raw
material classification”, In Proc. BMVC2017, 2017.
[15] L. Wolﬀ and T. Boult, “Constraining object featuresusing a polarization reflectance model”, IEEE Trans.
PAMI, 13(7), pp.635–657, 1991.
[16] R. Woodham, “Photometric method for determining
surface orientation from multiple images”, Optical Engineering, 19(1), pp.139–144, 1980.
[17] W. Ye, X. Li, Y. Dong, P. Peers, and X. Tong, “Single photograph surface appearance modeling with selfaugmented CNNs and inexact supervision”, Computer
Graphics Forum, 37(7), pp.201–211, 2018.
[18] H. Zheng, L. Kong, and S. Nahavandi, “Automatic inspection of metallic surface defects using genetic algorithms”, Journal of Materials Processing Technology,
125–126, pp.427–433, 2002.