NOLTA, IEICE
Paper

Progressive image transmission based onimage spatio-temporal decomposition bysigma-delta cellular neural network
Hisashi Aomori 1 a) , Ryohei Mizutani 2 , Hideharu Toda 1 ,and Tsuyoshi Otake 3
1

Faculty of Engineering, Chukyo University
101–2 Yagoto-honmachi, Showa-ku, Nagoya, Aichi, 466–8666, Japan

2

Yamazaki Mazak Corporation
1–131 Takeda, Oguchi-cho, Niwa-gun, Aichi, 480–0197, Japan

3

College of Engineering, Tamagawa University
6–1 Tamagawagakuen, Machida-shi, Tokyo, 194–0041, Japana)aomori@ieee.org

Received October 17, 2021; Revised December 16, 2021; Published April 1, 2022
Abstract: In this paper, we propose a novel progressive image transmission framework basedon spatio-temporal image decomposition and synthesis by the SD-CNN. In our method, weredesign the baseline SD-CNN and the weighted sum is introduced in the accumulator. This
innovation enables lossless or near-lossless progressive image transmission. Experimental resultsin various test images support that the image reconstruction performance of the SD-CNN has
dramatically improved by our method.
Key Words: spatio-temporal image decomposition, cellular neural network, progressive imagetransmission

1. Introduction
The progressive image transmission technique is widely used in remote sensing, medical images, andtelemedicine applications. In the progressive image transmission, an input image is transformed into
an image sequence. During reconstruction, a coarse approximation is ﬁrstly reconstructed by receiving a bit stream of early stages, then the quality of the reconstructed image is gradually enhanced byadding subsequent sequence. Also, progressive image transmission is often desired for band-limited
communication channels since it can reduce the number of bits per transmission. Traditional progressive image transmission can be categorized into two approaches [1–4]. One is a multi-resolution or
hierarchical coding method with the discrete wavelet transform (DWT). Most of this framework incorporates image compression methods. Especially, the JPEG 2000 and the SPIHT [5] are well-known
implementations. The other is the bit-plane method (BPM) that is the simplest implementation ofa progressive image transmission. This method utilizes the bit-plane decomposition. A n-bit im-

264
Nonlinear Theory and Its Applications, IEICE, vol. 13, no. 2, pp. 264–270©IEICE 2022

DOI: 10.1587/nolta.13.264

age is decomposed into n bit-planes, where i-th bit-plane is given by the i-th bit of binary numbers
representing a pixel value. Meanwhile, a few bio-inspired approaches have been studied. [6] is awell-known implementation, where radial basis function (RBF) networks are used as an interpolator
for image coding scheme. A majority of bio-inspired approaches are used in image coding frameworks,
while only a few methods achieve progressive image transmission framework utilizing a nature of thebiological system.
In the visual system, light is captured by the eyes, and its luminance value is converted to spatiotemporal spikes by the retina. This functionality suggests that the visual system has the potentialof spatio-temporal image decomposition which is applicable to progressive image transmission. In[7], we proposed the sigma-delta cellular neural network (SD-CNN) which enables a spatial domain
sigma-delta modulation. Like a sigma-delta modulation (SDM), a luminance value of each pixel isrepresented by pulse density modulated bitstream. Since the CNN is also a mimic of the retina,
the essential functionality of the SD-CNN can be thought of a spatio-temporal decomposition of animage. We experimentally showed that the SD-CNN can decompose a still image into a binary image
sequence and the quality of the reconstructed image is gradually enhanced by adding binary image
sequence. However, the development of the SD-CNN model that achieves high ﬁdelity progressive
image transmission still remains an open research issue.
In this paper, we propose a spatio-temporal image decomposition for progressive image transmission by the SD-CNN. In our method, from the ﬁndings of the image compression framework using
CNNs [8], we redesign the SD-CNN to improve image reconstruction performance. This innovation
has dramatically improved the image reconstruction performance and enabled lossless or near-lossless
progressive image transmission. Furthermore, by taking the weighted sum of an output binary sequence, we improve the eﬃciency of the accumulator which recovers the input image from a binary
image sequence output by the SD-CNN. Since progressive image transmission by our method consistsof pulse density modulation of an image and its demodulation, the mechanism of our approach iscompletely diﬀerent from conventional methods which utilize the DWT and the BPM. The encoding
and decoding processes for progressive image transmission in the proposed method are as follows.
In the encoder, an input image is pulse density modulated by the SD-CNN, and we can obtain one
binary image at each iteration of the SD-CNN dynamics. Then the output binary image sequence istransmitted to the decoder. In the decoder, the input image is restored via the accumulator, where
the weighted sum of each image in the binary image sequence is computed. This image recovery
process is inspired by the composition process of the BPM. Experiments on various grayscale test
images conﬁrm that the proposed method has lossless or near-lossless progressive image transmission
performance.
The rest of this paper is organized as follows. In section 2, a novel high ﬁdelity progressive imagetransmission framework using the SD-CNN is proposed. Details of redesigning the SD-CNN is given,and the deﬁnition of the new accumulator used in our method is described. Experiments on progressive
image transmission performance and eﬃciency are presented in section 3. Finally, some concluding
remarks are made in section 4.

2. Spatio-temporal image decomposition and progressive recovery bysigma-delta cellular neural network
Figure 1 shows the encoder and the decoder of our proposed progressive image transmission framework. A cell which is a processing unit of CNN and a pixel of the input image have a one-to-one
correspondence. Each cell is locally connected to its neighbors called r-neighborhood indicated by
the red rectangle in Fig. 1. Also, pixel values are converted to real numbers between -1 and 1 thatare corresponding to the dynamic range of CNN state variables.
In the encoder, an input image is decomposed into a spatio-temporal binary image sequence by
the dynamics of the SD-CNN. As shown in Fig. 2, we can obtain one binary image at each iterationof the SD-CNN dynamics. In our method, we utilize the ﬁndings of CNN-based image compression
framework [8] and the SD-CNN is redesigned for high ﬁdelity image reconstruction. The ﬁrst few

265

output images of the proposed method are very similar to that of the BPM indicated in Fig. 3. The
output sequence is transmitted to the decoder.
Encoder(Spatio-Temporal Image Decomposition by SD-CNN)

Decoder(Image Reconstruction by Spatio-Temporal Synthesis)

ΣInput Image

SD-CNN

Accumulator

Binary Image
Sequence

Gaussian
Filtering

Progressive
RecoveryFig. 1: Proposed system(a) 1st output(b) 2nd output(c) 3rd output(d) 4th output

Fig. 2: Example of output sequence of our method(a) 8th plane(b) 7th plane(c) 6th plane(d) 5th plane

Fig. 3: Example of output sequence of the BPM
In the decoder, the original image is recovered progressively shown in Fig. 4 from a binary image
sequence by spatio-temporal synthesis. The processes of spatio-temporal synthesis is very simple as
described below. Each binary image becomes an input to the accumulator and its weighted sum is
calculated. To determine the weight for each binary image, we utilize the weighting of the bit-plane
restoration as a reference. That is, they are set so that the ﬁrst output has the heaviest weight andgradually decreases. Finally, the reconstructed image is obtained by applying the Gaussian ﬁlter tothe weighted sum of the binary image sequence.

(a) 1st recovered image(b) 2nd recovered image(c) 3rd recovered image(d) 4th recovered image

Fig. 4: Example of progressive image recovery by proposed method

2.1 The SD-CNN for high ﬁdelity image reconstruction
The block diagram of a cell of the SD-CNN is illustrated in Fig. 5. The state equation of a cell at
the coordinates (i, j) is given by

266

ξ
uij

Z-1

xij(t +1)yij(t)
xij(t)A( i, j ; k, l )Fig. 5: The block diagram of a cell of the SD-CNN∑

xij (t + 1) = ξxij (t) +A(i, j; k, l)ykl (t) + uij ,(1)

C(k,l)∈Nr (i,j){

yij (t) = f (xij (t)) =1

xij (t) ≧ 0,

(2)

−1 otherwise,

where xij (t), yij (t), uij , f (·), A(i, j; k, l) and ξ are the internal state, the output, the input of a cell,the nonlinear output function, the A-template and the C-template, respectively. The r-neighborhood
Nr (i, j) is deﬁned by Nr (i, j) = {C(k, l)| max{|k − i|, |l − j|} ≤ r}. The A-templates is deﬁned by()
1(k − i)2 + (l − j)2
exp −
,
(3)
A(i, j; k, l) = −
2πσ 2
2σ 2
where σ is a standard deviation of the Gaussian function.
In the baseline SD-CNN [7], the A-template, the C-template, and σ are empirically determinedparameters. Especially, an optimal σ depends on an input image. In our method, these parameters
are determined by referring to the design policy in [8]:Design policy (i)The A-template is designed so that its center A(i, j; i, j) is −1.
Design policy (ii)
The coeﬃcient for xij (t) is deﬁned to be zero.
Note that the following points regarding the design policy (i) and (ii). Although the convergencecondition of the CNN is A(i, j; i, j) = 0, in our method, a spatio-temporal image decomposition isachieved by unsatisfying the convergence condition. For this reason, the proposed method adopts
design policies. In [8], once the A-template is designed so that its center A(i, j; i, j) is −1, thenA(i, j; i, j) is set to 0. The design policy (i) adopts the original design rule. A(i, j; i, j) becomes a
coeﬃcient of xij (t), if xij (t) = yij (t). So, the design policy (ii) adopts xij (t) = 0 instead of theconvergence condition.
From the design policy (i) and Eq. (3), we get 2πσ 2 = 1. This means that σ is independent of an
input image. For the design policy (ii), we assume that the output function f (·) is a piecewise linear
function deﬁned by f (x) = min(1, max(−1, x)) and x is in its linear region. Then, the state equationof the SD-CNN is rewritten as∑
xij (t + 1) = ξxij (t) + A(i, j; i, j)xij (t) +A(i, j; k, l)ykl (t) + uij(4)
C(k,l)∈Nr (i,j)(k,l)̸=(i,j)= (ξ + A(i, j; i, j)) xij (t) +∑A(i, j; k, l)ykl (t) + uij .

(5)C(k,l)∈Nr (i,j)(k,l)̸=(i,j)

To satisfy the design policy (ii), we have ξ + A(i, j; i, j) = 0 from Eq. (5). Therefore, ξ is determinedas 1 since A(i, j; i, j) is -1.

2.2 Progressive image reconstruction by spatio-temporal synthesis
The output of the SD-CNN becomes the input of the decoding processes. The spatio-temporal synthesis for image recovery consists of two parts. At the ﬁrst, the accumulator calculates a weighted

267

sum of the time series of each pixel. To determine the weight for the t-th image, we use the weightingof the bit-plane restoration as a reference. Let Bi be the i-th plane image of the BPM for n bit image,∑nthen reconstruction image I is given by I = i=1 2i−1 Bi . In our method, we set the weights so thatthe ﬁrst output has the heaviest weight and gradually decreases. A weighted sum of the time seriesof each pixel sij (t) is given bysij (t) =t∑(t + 1 − m)yij (m)

(6)

m=1

Finally, the reconstructed image at t-th iteration ũij (t) is obtained by applying the D-template.
∑D(i, j; k, l)skl (t),(7)
ũij (t) =C(k,l)∈Nr (i,j))(
1(k − i)2 + (l − j)2
D(i, j; k, l) =
exp −
.
2πσ 2
2σ 2

(8)

3. Experimental Results

PSNR for Average MSE [dB]120

Proposed Method# of Lossless Reproduction
Baseline
SDM

100

8

80

6

60

4

40
2

20
0

0

50

100

150

200

250

Number of Lossless Reproduction

To evaluate the eﬀectiveness of the proposed method, we perform image coding/decoding experiments
on standard grayscale images. An example of a coded image sequence and its decoding processes areillustrated in Figs. 2 and 4. In all experiments, the r-neighborhood of the SD-CNN is set to r = 2,
and the maximum number of iterations is set to 256. The peak signal to noise ratio (PSNR) for an
average of the mean square error (MSE) is employed for objective performance criteria. Since some
images can be recovered without loss by the proposed method, the average PSNR cannot be calculated(PSNR value for lossless reconstruction is inﬁnity). Hence, PSNR for average MSE is employed foran objective image quality measurement. Generally, if PSNR is higher than 40 dB, two images areperceptually indistinguishable (visually lossless). Moreover, if the PSNR is higher than 50 dB, thedistortion between the reconstructed image and the original image reaches a near-lossless quality.
First, the progressive image transmission performance of our method is compared with the baseline
SD-CNN [7] and the ﬁrst-order SDM. A pixel and the SDM have a one-to-one correspondence. Therefore luminance of each pixel becomes an input to the corresponding SDM and an average output of aperiod implies recovered luminance. In this experiment, we use 9 standard grayscale images indicatedin Fig. 6 from the dataset of [7]. Resolution of all images is 512 × 512.

0

Iterations

Fig. 6: Test images from the dataset of [7]Fig. 7: Progressive image transmission performance ofeach method

Progressive image transmission performance at each iteration is summarized in Fig. 7. In thisﬁgure, the blue curve means the image reconstruction performance of our method, the purple curveis that of the baseline SD-CNN, and the green curve is that of the SDM. The pink curve shows thenumber of images that are reproduced losslessly by our method. From this ﬁgure, we can see that the268

reconstruction performance and eﬃciency of our method are much better than that of the baseline
SD-CNN and the SDM. In particular, only our method can recover all test images losslessly.
Our method achieves over 40 dB PSNR by only 12 iterations, and the PSNR of the baseline method
is always below 40 dB. The proposed method can achieve the same reconstruction performance of thebaseline SD-CNN by only 8 iterations. On the other hand, the SDM requires 58 iterations to achieve
the same performance as our method.
Next, we evaluate the performance of the proposed method and the SDM on a high-deﬁnition image
dataset. In this experiment, we use the Kodak photo CD [9] dataset1 shown in Fig. 8.
Progressive image transmission performance at each iteration is summarized in Fig. 9. In thisﬁgure, the blue curve means the image reconstruction performance of our method, and the green
curve is that of the SDM. The pink curve shows the number of images that are reproduced losslessly
by our method. From this ﬁgure, we can see that the progressive image transmission performance
improves in proportion to the number of iterations. We also ﬁnd that not all images can be restored
losslessly. The PSNR curve of the SDM towards around 50 dB by 256 iterations, meanwhile, ourmethod can achieve the same performance by only 33 iterations. Therefore, the reconstructed image
by the proposed method quickly reaches a near-lossless quality.

Fig. 8: The Kodak image dataset [9]Proposed Method# of Lossless Reproduction
SDM

0

50

100

150

Iterations

200

250

24
22
20
18
16
14
12
10
8
6
4
2
0

110

PSNR for Average MSE [dB]

110
100
90
80
70
60
50
40
30
20
10
0

Number of Lossless Reproduction

PSNR for Average MSE [dB]To evaluate the importance of each image in an image sequence, the starting position of the sequenceis varied from 1 to 256 while keeping the sequence length of 256. Experimental results are summarized
in Fig. 10 where the blue curve shows image reconstruction performance. We can see that the graphdrops rapidly and settles around 50 dB. This indicates that the ﬁrst output of the SD-CNN has great
importance. We believe that this PSNR characteristic is reasonable because the ﬁrst output of theSD-CNN and the bit plane of the most signiﬁcant bit in the BPM which plays the most importantrole for the bit-plane synthesis, are very similar. Also, we can see that if the sequence length is longenough, very high image restoration performance can be obtained.
Proposed Method

100
90
80
70
60
50
40

0

50

100

150

200

Starting Point of Accumulation

250

Fig. 9: Progressive image transmission perfor- Fig. 10: Evaluation of the importance of each immance for the Kodak dataset
age in an image sequence
1
This dataset consists of 24 color images and their resolution is 768×512. In this experiment, these images are converted
from the RGB color space to the ITU-R BT.601 YCbCr color space, and the Y images are used for the grayscale test
images.

269

4. Conclusion
In this paper, a novel progressive image transmission framework based on spatio-temporal image
decomposition and synthesis by the SD-CNN has been proposed. In our method, the baseline SDCNN is redesigned and the weighted sum is introduced in the accumulator. Experimental results invarious test images support that this innovation has dramatically improved the image reconstruction
performance and enabled lossless or near-lossless progressive image transmission. It is also found thatthe ﬁrst output of the SD-CNN is most signiﬁcant for image reconstruction.

Acknowledgments
This research was partially supported by a research grant (2020001) of Hagiwara Foundation, and
JSPS KAKENHI grant number 21K12052.

References[1] K. Hung, C. Chang, and I. Lin, “Lossless compression-based progressive image transmission
scheme,” The Imaging Science Journal, vol. 52, no. 4, pp. 212–224, 2004. DOI: 10.1179/
136821904225020276[2] R. Dilmaghani, A. Ahmadian, M. Ghavami, and A. Aghvami, “Progressive medical image transmission and compression,” IEEE Signal Processing Letters, vol. 11, no. 10, pp. 806–809, 2004.
DOI: 10.1109/LSP.2004.835563[3] J. M. Zheng, D. W. Zhou, and J. L. Geng, “ROI progressive image transmission based on wavelet
transform and human visual specialties,” Proc. of ICWAPR’07, vol. 1, pp. 260–264, 2007. DOI:
10.1109/ICWAPR.2007.4420675[4] H. Ravikiran and Paramesha, “A hybrid progressive image compression transmission and reconstruction architecture,” Lecture Notes in Electrical Engineering, vol. 545, pp. 867–875, 2019.
DOI: 10.1007/978-981-13-5802-9 75[5] A. Said and W. A. Pearlman, “A new fast and eﬃcient image codec based on set partitioning inhierarchical trees,” IEEE Trans. Circuits Syst. Video Technol., vol. 6, no. 3, pp. 243–250, Jun.
1996. DOI: 10.1109/76.499834[6] T. Sigitani, Y. Iiguni, and H. Maeda, “Image interpolation for progressive transmission by using
radial basis function networks,” IEEE Trans. Neural Netw., vol. 10, no. 2, pp. 381–390, Mar.
1999. DOI: 10.1109/72.750567[7] H. Aomori, T. Otake, N. Takahashi, and M. Tanaka, “Sigma-delta cellular neural network for 2d modulation,” Neural Networks, vol. 21, pp. 349–357, Mar. 2008. DOI: 10.1016/
j.neunet.2007.12.020[8] Y. Kawai, H. Toda, H. Aomori, T. Otake, I. Matsuda, and S. Itoh, “Hierarchical lossless image compression using cnn predictors optimized by adaptive diﬀerential evolution and contextadaptive coding,” Proc. of NOLTA’20, pp. 65–67, Nov. 2020.
[9] “Kodak lossless true color image suite,” http://r0k.us/graphics/kodak/.

270