PROCEEDINGS of the
23rd International Congress on Acoustics
9 to 13 September 2019 in Aachen, Germany

Sound capture from rolling-shuttered visual camera
based on edge detection
Koichi TERANO1; Hiroki SHINDO2;
Kenta IWAI3; Takahiro FUKUMORI4; Takanobu NISHIURA5
1, 2

Graduate School of Information Science and Engineering, Ritsumeikan University, Japan
3, 4, 5

College of Information Science and Engineering, Ritsumeikan University, Japan

ABSTRACT
Recently, several new studies for recording sound have been conducted on extracting sounds from images ofan object surface vibrated by sound waves. This method can capture the sound by a high-speed camera insteadof an air conduction microphone. However, this method is unpractical due to higher cost of the high-speed
camera. In this paper, we propose a new sound capture method with lower cost using a rolling-shutteredvisual camera. This camera uses a CMOS (complementary metal-oxide-semiconductor) image sensor andsequentially writes an image from the top row to the bottom row. Therefore, when a moving object isphotographed, a rolling-shutter distortion occurs due to the different writing time for each row. The proposed
method uses the edge detection to emphasize the edge of the rolling-shutter distortion caused byphotographing the vibrating object. Then, the proposed method resamples the edge as the amplitude of thesound wave. As a result of the sound capturing experiment, we confirmed that the proposed method cancapture pure tone sound below 1,000 Hz from the image photographed by the CMOS image sensor.
Keywords: Sound capture, Rolling-shuttered visual camera, CMOS image sensor, Edge detection

1. INTRODUCTION
A typical air conduction microphone captures not only the target sound but also the noise emitted
from the sound sources surrounding of the microphone. Recently, for the purpose of only target soundcapture, several sound capture methods by measuring the vibration of objects affected by sound waves
have been proposed [1, 2, 3]. In [1], a laser Doppler vibrometer is used and captures sound byirradiating a vibrating object with laser light and measuring difference between the reflected light andthe reference light. In [2, 3], a high-speed camera is used and directly captures sound by photographing
the vibration of the object and measuring the displacement of surface. In these methods, it is possible
to capture only the target sound by measuring the vibration of the object near the target sound source.
However, these methods are unpractical due to high cost of the laser Doppler vibrometer or that of
the high-speed camera.
We propose a new sound capture method from the image of the vibrating object affected by soundwaves with a rolling-shuttered visual camera. This camera is widely used in digital cameras andsmartphone cameras [4]. Thus, the cost of the proposed method is lower than that of the conventional
methods. This camera uses a small image sensor called complementary metal-oxide-semiconductor(CMOS) image sensor that senses light and converts it into an electrical signa l. The CMOS image
sensor sequentially writes an image from the top row to the bottom row. Therefore, whenphotographing a moving object, rolling-shutter distortion occurs in the image due to the difference inphotographing time between each row [5]. In this paper, we use the rolling-shuttered visual camera to
photograph the surface of a vibrating object. The proposed method uses the edge detection toemphasize the edge of rolling-shutter distortion caused by sound waves, and measures displacement1

is0267rs@ed.ritsumei.ac.jp
is0261xf@ed.ritsumei.ac.jp
3
iwai18sp@fc.ritsumei.ac.jp
4
fukumori@fc.ritsumei.ac.jp
5nishiura@is.ritsumei.ac.jp
22878

of vibration to capture the sound affecting the object. We conducted an experiment to confirm theperformance of the proposed method.

2. Image capture with CMOS image sensor
In this paper, we use the rolling-shuttered visual camera to photograph the surface of a vibrating object
affected by sound waves. This camera is widely used in digital cameras that built into smartphone becauseof its small size and low power consumption [4]. It employs a CMOS image sensor as the imaging device.
When the moving object is photographed by this sensor, rolling-shutter distortion occurs due to the differencein photographing time between each row. The principle of the CMOS image sensor and rolling-shutter
distortion are shown in Fig. 1. Photographing with CMOS image sensor is performed by energizing andexposing a photodiode disposed in a pixel and converting received light into electrical signal. In order toreduce the power consumption at photographing, the CMOS image sensor performs exposure by energizing
the photodiodes sequentially from the top row to the bottom row. Therefore, when photographing a moving
object, the position of the object changes during exposure from row to row. This positional deviation is called
rolling-shutter distortion [5]. Sound waves propagate to the environment as air pressure fluctuations. Hence,the surface of the object near the sound source vibrates due to affection of the sound wave. The exposure
sampling frequency of the CMOS image sensor is sufficiently higher for human audible sound. As a result,
the rolling-shuttered visual camera can photograph surface vibration of the object affected by sound waves
as rolling-shutter distortion. In other words, we can capture the sound affecting objects by detecting therolling-shutter distortion caused by sound waves on the image of vibrating object.

3. Sound capture based on edge detection from the image
In this paper, we propose a new sound capture method by measuring the displacement of the vibrating
object affected by sound waves based on the edge detection for the rolling-shutter distortion. For simply
explanation, we assume the situation of photographing straight lines on a paper in this section. The overview
of the proposed method is shown in Fig. 2. The photographed image is stored as a grayscale image source ofa matrix having only light intensity values 0-255 for each pixel. At first, the proposed method detects theedge of the grayscale image using Canny operator [6] which is one of the most widely used because of its
robustness to various image size [7, 8]. The edge detection using Canny operator has four steps as follows.

Figure 1 – Principle of CMOS image sensor and rolling-shutter distortion.

2879

Step 1: Low pass filtering with Gaussian filter
Gaussian smoothing filter is used to remove impulse noise on the image. In this paper, we use the ͵ ൈ ͵neighborhood kernel which is one of the most widely used for Gaussian filter. The ͵ ൈ ͵ neighborhood
image source ࢂ௫ǡ௬ and the kernel of Gaussian filter ۵ are respectively shown in Eqs. (1) and (2). The
smoothed image ‫ݏ‬௫ǡ௬ by Gaussian filter is given by Eq. (3).
‫ݒ‬௫ିଵǡ௬ିଵ ‫ݒ‬௫ǡ௬ିଵ ‫ݒ‬௫ାଵǡ௬ିଵ‫ݒ‬௫ǡ௬
‫ݒ‬௫ାଵǡ௬ ൩ǡࢂ௫ǡ௬ ൌ ൥ ‫ݒ‬௫ିଵǡ௬
(1)‫ݒ‬௫ିଵǡ௬ାଵ ‫ݒ‬௫ǡ௬ାଵ ‫ݒ‬௫ାଵǡ௬ାଵͳ ͳ ʹ ͳ۵ൌ(2)
൥ʹ Ͷ ʹ൩ǡͳ͸ͳ ʹ ͳ‫ݏ‬௫ǡ௬ ൌ ࢂ௫ǡ௬ ‫ כ‬۵ǡ
(3)
ሺ‫ݔ‬ǡ
where ‫ݒ‬௫ǡ௬ is a pixel value of the input image at coordinate‫ݕ‬ሻ and symbol ‘‫ ’כ‬is a convolution operator.
Step 2: Image gradient calculation with Sobel filter.
Sobel filter is used to calculate the gradient and gradient direction of each pixel. Each directional kernel ofSobel filter ۵୶ ǡ ۵୷ is shown in Eq. (4).
െͳ Ͳ ͳെͳ െʹ െͳ۵୶ ൌ ൥െʹ Ͳ ʹ൩ ǡ ۵୷ ൌ ൥ Ͳ
(4)
Ͳ
Ͳ ൩Ǥ
െͳ Ͳ ͳͳʹͳThe gradient ‫݌‬௫ǡ௬ and the gradient direction ߠ௫ǡ௬ of each pixel are given by Eqs. (5) and (6).
ଶଶ‫݌‬௫ǡ௬ ൌ ට൫ࡿ௫ǡ௬ ‫ כ‬۵୶ ൯ ൅ ൫ࡿ௫ǡ௬ ‫ כ‬۵୷ ൯ ǡߠ௫ǡ௬ ൌ ିଵ ቆࡿ௫ǡ௬ ‫ כ‬۵୷ቇǡࡿ௫ǡ௬ ‫ כ‬۵୶where ࡿ௫ǡ௬ is the ͵ ൈ ͵ neighborhood smoothed image.

Figure 2 – Overview of the proposed method.

2880

(5)
(6)

Step 3: Non-maximum suppression (NMS)
NMS is adopted to obtain the accurate positioning and refinement edge. If the target pixel’s gradient is not
maximal value among the three pixels adjacent in the gradient direction ߠ௫ǡ௬ , the gradient is set to 0. The
gradient image adopted NMS‫݌‬Ԣ௫ǡ௬ is given by Eq. (7).
ͳͳ͹͹൬ǣ െ ߨ ൑  ߠ௫ǡ௬  ൏  ߨ ‫ߠ  ׫‬௫ǡ௬  ൐  ߨ ‫ߠ  ׫‬௫ǡ௬  ൏  െ ߨ൰ͺͺͺͺ‫݌‬௫ǡ௬ ‫݌‬௫ǡ௬ ൐ ‫݌‬௫ିଵǡ௬ ‫݌ ת‬௫ǡ௬ ൐ ‫݌‬௫ାଵǡ௬ǡ
‫݌‬Ԣ௫ǡ௬ ൌ ൜Ͳͳ͵͹ͷ൬ǣ ߨ ൑  ߠ௫ǡ௬  ൏  ߨ ‫  ׫‬െ ߨ ൑  ߠ௫ǡ௬  ൏  െ ߨ൰ͺͺͺͺ‫݌‬௫ǡ௬ ‫݌‬௫ǡ௬ ൐ ‫݌‬௫ାଵǡ௬ିଵ ‫݌ ת‬௫ǡ௬ ൐ ‫݌‬௫ିଵǡ௬ାଵ ‫݌‬Ԣ௫ǡ௬ ൌ ൜ǡͲ͵ͷͷ͵൬ǣ ߨ ൑  ߠ௫ǡ௬  ൏  ߨ ‫  ׫‬െ ߨ ൑  ߠ௫ǡ௬  ൏  െ ߨ൰ͺͺͺͺ‫݌‬௫ǡ௬ ‫݌‬௫ǡ௬ ൐ ‫݌‬௫ǡ௬ିଵ ‫݌ ת‬௫ǡ௬ ൐ ‫݌‬௫ǡ௬ାଵ ǡ
‫݌‬Ԣ௫ǡ௬ ൌ ൜Ͳͷ͹͵ͳ൬ǣ ߨ ൑  ߠ௫ǡ௬  ൑  ߨ ‫  ׫‬െ ߨ ൑  ߠ௫ǡ௬  ൏  െ ߨ൰ͺͺͺͺ‫݌‬௫ǡ௬ ‫݌‬௫ǡ௬ ൐ ‫݌‬௫ିଵǡ௬ିଵ ‫݌ ת‬௫ǡ௬ ൐ ‫݌‬௫ାଵǡ௬ାଵ ǡ‫݌‬Ԣ௫ǡ௬ ൌ ൜Ͳ

(7)

Step 4: Hysteresis threshold
The hysteresis threshold is adopted to check and connect edges. The hysteresis threshold binarizes the image
with two thresholds ܶ୫୧୬ and ܶ୫ୟ୶. If the gradient ‫݌‬Ԣ௫ǡ௬ is larger than ܶ୫ୟ୶, its pixel value ‫ݒ‬Ԣ௫ǡ௬ is set to
255 as the edge. If it is smaller than ܶ୫୧୬ , its pixel value is set to 0 as the non-edge. The pixels that gradient‫݌‬Ԣ௫ǡ௬ is between the two thresholds are set to 255 as long as there is at least one edge pixel around it;otherwise they are set to 0. From these steps, a binarized image ࢂԢ is obtained by Canny operator. The
proposed method adopts resampling and standardization to capture sound from the binarized image ࢂԢ.
Resampling
In the binarized image ࢂԢ, ‫ ݔ‬coordinates indicate displacement of surface vibrations and ‫ ݕ‬coordinates
indicate time index ݊ of the acquired signal. Thus, the proposed method stores ‫ ݔ‬coordinates to edge ofeach row sequentially as the amplitude of the acquired signal. Then, resampling is performed according tosampling frequency of the acquired signal and that of the CMOS image sensor. The acquired signal ܽሺ݊ሻ isgiven by Eq. (8).
ᇱܽሺ݊ሻ ൌ   ‫࡯ ݔ‬ǣ ‫ݒ‬௫ǡ௬ൌ ʹͷͷǡ ‫ ݕ‬ൌ௫ఢ࡯‫ܨ‬େ୑୓ୗ
‫ܨ‬୭୳୲ൈ ݊ ൬Ͳǡ ǥ ǡ ݊ǡ ǥ ǡൈ ൰ǡ‫ܨ‬୭୳୲
‫ܨ‬େ୑୓ୗ

(8)

where ݊ is time index of the acquired signal, ‫ܨ‬୭୳୲ is the sampling frequency of the acquired signal, ‫ܨ‬େ୑୓ୗis that of CMOS image sensor, ‫ݒ‬Ԣ௫ǡ௬ is a pixel value of the binarized image ࢂԢ, and height is max value of‫ ݕ‬coordinate.
Standardization
The proposed method can capture acquired signal from the image by resampling. However, as shown in Fig.
2, all amplitudes of the acquired signal are positive values because these are calculated as the amount ofdeviation from the y-axis of the image. Therefore, it is necessary to convert the amplitude average to 0. The
proposed method divides the acquired signal into short time frames and performs standardization so that theamplitude average of each frame is 0. By standardizing for each short time frame, it reduces low frequency
noise included due to factors such as paper inclination. The lth frame of the captured sound ‫ݏ‬ሺ݊ǡ ݈ሻ is given
by Eq. (9).
‫ݏ‬ሺ݊ǡ ݈ሻ ൌ ܽሺ݊ǡ ݈ሻ െ ܽതሺ݈ሻǡ(9)
where ܽതሺ݈ሻ is average of the acquired signal at ݈th frame. We can obtain the sound that vibrates the object
as the captured sound‫ݏ‬ሺ݊ሻ by connecting frame ‫ݏ‬ሺ݊ǡ ݈ሻ.

2881

4. Sound capture experiment
4.1

Experimental conditions
We carried out an experiment to evaluate the performance of the proposed method. The
experimental arrangement and vibrating object are shown in Fig. 3. The experimental conditions areshown in Table 1. In this experiment, we used a half colored paper of A4-size as a vibrating object,
and the boundary of the colored paper was located in 10 mm front of the loudspeaker. In addition, inorder to obtain sufficient illuminance, a light projector was used as a light source because a CMOS
image sensor in a dark place generates a lot of noise in the image. The experiment was conductedunder the light source. As sound sources, pure tone sounds with 400 Hz, 1,000 Hz, and 1,600 Hz wererespectively used and emitted from the loudspeaker. In this experiment, the height of each image
source is 1,080 pixels and the sampling rate of the CMOS image sensor is 48,000 Hz. This means the
proposed method can capture the sound of 25 ms length from one image source.
4.2

Experimental result
Image sources and binarized images are respectively shown in Figs. 4 and 5. In this section, forsimply explanation, image sources and binarized images are displayed rotated 90 degrees and trimmed.
Thus, the vertical direction indicates the displacement of the vibrating object and the horizontal
direction indicates the time information by the CMOS image sensor. From Fig. 4, the amplitude ofvibration affected by higher frequency sound is smaller than that of lower frequency sound. That is
because an object with mass has a small displacement by a higher frequency sound. For this reason,
from Fig. 5, edge detection accuracy of higher frequency sound is lower than that of lower frequency
sound. It is considered that the captured sound distorts due to low edge detection accuracy.

Figure 3 – Experimental arrangement and vibrating object.
Table 1 – Experimental conditions.
Environment(ܶ଺଴ ൌ ͳͲͲ)Background noise‫ܮ‬୅ ൌ ʹ͹Ǥͷ

Temperature / HumidityͳͻǤͲԨ / ͵͹Ǥ͸ΨSound source

2882Soundproof room

Pure toneሺͶͲͲ ǡ ͳǡͲͲͲ ǡ ͳǡ͸ͲͲ ሻ

Sound pressure level‫ܮ‬௔ ൌ ͳͳͲͲ

Sampling frequency / bitrateͺǡͲͲͲ  / 16 bits

Rolling-shuttered camera

Canon, EOS 5D Mark,,

Lens

Canon, MP-E 65 mm f/2.8 1-5xSize of image sourceͳǡͻʹͲ ൈ ͳǡͲͺͲLight source

LED light (10,000 lm)

Waveforms and power spectra of each captured sound are respectively shown in Figs. 6 and 7.
From Fig. 6, as we expected, distortion of higher frequency sound is larger than that of lower
frequency sound. From Fig. 7, we can confirm the peak at contained frequency of each sound. In otherwords, the proposed method can capture the sound from the image source. Furthermore, the SNR of
the captured sound with higher frequency sound is worse than that of lower frequency sound due todistortion of its waveform. In particular, for the pure tone with 1,600 Hz, the power of the captured
sound is lower than that of noise. For that reason, when capturing sound which contains frequencies
above 1,600 Hz, it is necessary to use a high-zoom camera or a lens to photograph the sufficient
displacement for detecting edges and reducing distortion. In addition, it is considered that performing
weighted summation at multiple locations for edge detection is useful to emphasize the higher
frequency sound. From these results, the proposed method can capture sound from the image
photographed by the rolling-shuttered visual camera and the problems are clarified.

Figure 4 – Image source.

Figure 5 – Binarized image.

Figure 6 – Waveform of captured sound.

Figure 7 – Power spectra of captured sound.

2883

5. CONCLUSIONSIn this paper, we proposed the new sound capture method from the image of the vibrating object
affected by sound waves with a rolling-shuttered visual camera. The proposed method uses the edge
detection to emphasize the edge of rolling-shutter distortion caused by sound waves, and measuresdisplacement of vibration to capture the sound affecting the object. We conducted an experiment toconfirm the performance of the proposed method. From the experimental results, we confirmed thatthe proposed method can capture pure tone sound with 1,000 Hz from the image photographed by therolling-shuttered camera. In addition, we also confirmed that extracting higher frequency sound ismore difficult than lower frequency sound due to small displacement of vibration. In the future, we
intend to utilize acoustics signal processing to emphasize higher frequency sound and movie sources
to capture various sounds.

ACKNOWLEDGEMENTS
This work was partly supported by JST-COI and JSPS KAKENHI Grant Numbers JP18K19829,
JP19H04142, and R-GIRO (Ritsumeikan Global Innovation Research Organization) funded by
Ritsumeikan University.

REFERENCES
1. J. Shan, Y. He, D. Liu and W Chen, "Laser Doppler Vibrometer for long-distance acoustical signals
acquirement," Conference on Lasers & Electro Optics & The Pacific Rim Conference on Lasers andElectro-Optics, pp. 1-2, 2009.
2. M. Hua, L. Zhou, C. Liu and Z. Li, "The research of vibration detection using the visual microphone
technology," 10th International Conference on Measuring Technology and Mechatronics Automation, pp.
256-258, 2018.
3. J. Ahn and D. Kim, "Simple and effective speech enhancement for visual microphone," 4th IAPR Asian
Conference on Pattern Recognition, pp. 694-699, 2017.
4. S. Baker, E. Bennett, S. B. Kang and R. Szeliski, "Removing rolling shutter wobble," IEEE Computer
Society Conference on Computer Vision and Pattern Recognition, pp. 2392-2399, 2010.
5. J. Chun, H. Jung and C. Kyung, "Suppressing rolling-shutter distortion of CMOS image sensors by motion
vector detection," IEEE Transactions on Consumer Electronics, vol. 54, no. 4, pp. 1479-1487, 2008.
6. J. Canny, "A computation approach to edge detection," IEEE Transactions on pattern analysis andmachine intelligence, vol. 8, no. 6, pp. 679-698, 1986.
7. A. P. Thombare and S. B. Bagal, "A distributed canny edge detector: Comparative approach,"International Conference on Information Processing, pp. 312-316, 2015.
8. C. Sun, X. Fan and D. Zhao, "A fast intra cu size decision algorithm based on Canny operator and SVM
classifier," 25th IEEE International Conference on Image Processing, pp. 1787-1791, 2018.

2884