ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) | 978-1-6654-0540-9/22/$31.00 ©2022 IEEE | DOI: 10.1109/ICASSP43922.2022.9747874

WEIGHTED WAVELET-BASED SPECTRAL-SPATIAL TRANSFORMS FOR CFA-SAMPLED
RAW CAMERA IMAGE COMPRESSION CONSIDERING IMAGE FEATURES
Liping Huang† and Taizo Suzuki‡†‡

Graduate School of Science and Technology, University of Tsukuba, Japan
Faculty of Engineering, Information and Systems, University of Tsukuba, Japan
Email: † huang@wmp.cs.tsukuba.ac.jp, ‡ taizo@cs.tsukuba.ac.jp
ABSTRACTTo efficiently compress raw camera images captured using a color
filter array (CFA-sampled raw images), wavelet-based spectralspatial transforms (WSSTs) that change a CFA-sampled raw image
from an RGB color space into a decorrelated color space have beenpresented. This study introduces weighted WSSTs (WWSSTs) thatwork especially for the CFA-sampled raw images with many edges
well. The WWSSTs are obtained by considering that each predict
step of the conventional WSSTs is constructed by a combinationof two 1-D diagonal transforms and by weighting them along theedge directions in the images. The experiments at JPEG 2000-based
lossless and lossy compression basically show that compared with
the WSSTs, our WSSTs improve the results for the images withmany edges by about 0.04 bpp in LBRs, 2.10 [%] in BD-rates, and0.12 dB in BD-PSNRs while keeping the compression efficiency forthe general images.
Index Terms— Color filter array, raw camera image, losslessand lossy compression, spectral-spatial transforms, wavelet transforms.
1. INTRODUCTION
RAW camera image is mainly obtained by placing a color filter array(CFA) between the light sensors and the camera lens. To economize
the cost of hardware, most cameras capture a color image with a single sensor instead of using three RGB sensors. Each sensor collects
a color component, which is composed of either red, green, or blue
component for each pixel, not full-color one, and the obtained raw
data is called CFA-sampled raw image. The most popular and widelyused CFA is the Bayer CFA (Fig. 1). Using the CFA-sampled raw
image as it is, an image processor performs most of the preprocessing such as white balancing, denoising, and demosaicing, which are
irreversible processes. Especially, the image quality largely depends
on the performance of the demosaicing process.
The demosaicing-first compression method, which demosaics
the CFA-sampled raw image before compression, has been employed as the standards such as JPEG and JPEG 2000. However,since the demosaicing-first one causes redundancy, i.e., data volume
of a full-color image is three times compared with one of the original
CFA-sampled raw image, the compression-first method that compresses the CFA-sampled raw image before demosaicing and avoids
the redundancy is gathering attention. In addition, the compressionfirst one allows us higher degree of freedom that performs various
image processing than the demosaicing-first one. Many CFA image
compression methods have been presented [1–9], all demonstrating
the efficiency of the compression-first scheme.

978-1-6654-0540-9/22/$31.00 ©2022 IEEE

Fig. 1. Bayer CFA.

This study focuses on spectral-spatial transforms (SSTs) [3–9].
They change a CFA-sampled raw image from an RGB color space
into a decorrelated color space, such as YDgCbCr or YDgCoCg
color space composed of luma, difference green, and two chroma
components. The spectral redundancy between the decorrelated
components is very small. Since the human visual system is notvery sensitive against the distortions of high frequency (different
green) and chroma components, the strong compression of the components will not affect the image quality much. However, the direct
use of wavelet transforms for the SSTs is nothing but ignoring theimage features, especially, edge information. On the other hand,(non-redundant) adaptive directional wavelet transforms [10–12],
which adapt the filtering directions to the orientations of edge information, are well-known as efficient methods for considering theimage features.
Inspired by the adaptive directional wavelet transforms, thisstudy introduces weighted WSSTs (WWSSTs) that work especially for the CFA-sampled raw images with many edges well.
The WWSSTs are obtained by considering that each predict stepof the conventional WSSTs to YDgCoCg color space (YDgCoCgWSSTs) in [8] is constructed by a combination of two 1-D diagonaltransforms and by weighting them along the edge directions in theimages. The experiments at JPEG 2000-based lossless and lossy
compression show that our WWSSTs achieve comparable performance to the conventional WSSTs in the case of the general images
and that our WWSSTs outperform the conventional WSSTs becauseof more efficient decorrelation in the case of the images with manyedges.
2. REVIEW AND DEFINITIONS
2.1. Wavelet Transforms
Cohen-Daubechies-Feauveau (CDF) wavelet transforms [13] arecommonly used for image processing. A 1-D wavelet transform is

1850

ICASSP 2022

Authorized licensed use limited to: Tsukuba Univ Lib. Downloaded on May 02,2022 at 08:15:30 UTC from IEEE Xplore. Restrictions apply.

whereTable 1. Coefficients of 5/3 and 9/7 wavelet transforms.
5/3
9/7
p0 −1/2 −1.58613434205992
u01/4−0.05298011857295
p1
00.882911075530940
u100.443506852043967


Tog = P2W(z2−1 )OW(z1−1 , z2 )
O
P1
I
O



O
W(z1−1 , z2−1 ).
(4)Here, P1 and P2 are permutation matrices:1
0 0 1 00
1 0 0 0 , P2 = P1 = 
0
0 1 0 00
0 0 0 1

0
0
0
1

0
1
0
0


0
0
.
1
0

(5)

A 2-D customized wavelet transform W(z1 , z2 ) is expressed by predict and update steps as follows:



0
Y
1 Uk (z1 , z2 )
1
0
, (6)
W(z1 , z2 ) =Pk (z1 , z2 ) 1
01
k=N −1 |{z}|{z}
update step

predict step

where z1 and z2 are horizontal and vertical delay elements, N is thenumber of iterations of the two lifting steps depending on the type ofwavelet transforms, and Pk (z1 , z2 ) and Uk (z1 , z2 ) are polynomials
with coefficients pk and uk :1
(1 + z1−1 + z2−1 + z1−1 z2−1 )pk ,(7)2
1
Uk (z1 , z2 ) = (1 + z1 + z2 + z1 z2 )uk ,(8)2
respectively. The actual implementation is shown in Fig. 3. The pixels used in predict and update steps are not limited within a focused
2×2 macropixel and the surrounding pixels are also used to makethe predict results more accurate. However, due to not considering
the image features, the prediction may not be so accurate in somecases.
Pk (z1 , z2 ) =Fig. 2. Weights: (top-left) w1n for G2 prediction, (top-right) w2n
for G2 prediction, (bottom-left) w1n for R prediction, and (bottomright) w2n for R prediction.

3. WEIGHTED WAVELET-BASED SPECTRAL-SPATIALTRANSFORMS

expressed by predict and update steps as follows:
W(z) =

0Y
10
k=N −1 |Uk (z)1
1
Pk (z){z}|{z

update step


0
,
1
}

(1)

W1 (1 + z1−1 z2−1 ) + W2 (z1−1 + z2−1 )pk .
Pek (z1 , z2 ) =W1 + W2

predict step

where z is a delay element and Pk (z) and Uk (z) are polynomials
with coefficients pk and uk :Pk (z) = (1 + z −1 )pk

andThis study customizes the polynomial Pk (z1 , z2 ) in the predict stepsof the WSSTs into

Uk (z) = (1 + z)uk .

(2)

When W1 = W2 = 1, it is clear that Pek (z1 , z2 ) = Pk (z1 , z2 ).
Consequently, we can rewrite it for the case of each of G2 and R
predictions as−1

Table 1 shows the coefficients pk and uk in the 5/3 wavelet transforms (N = 1) and 9/7 wavelet transforms (N = 2). Since the 5/3
and 9/7 wavelet transforms can predict the pixels of interest moreaccurately than Haar wavelet transforms do, they can decorrelate thesignals more.
2.2. Wavelet-based Spectral-Spatial Transforms to YDgCoCg
Color Space−1

W1 (1 + z1 z2 ) + W2 (z1 + z2 )pk
Pek (z1−1 , z2 ) =W1 + W2
if G2 prediction,(10)

W1 (1 + z1 z2 ) + W2 (z1 + z2 )Pek (z1−1 , z2−1 ) =pk
W1 + W2
if R prediction,

(11)

where

This study focuses specifically on the YDgCoCg-WSSTs in [8],which we will hereafter simply refer to as the WSST. When R, G1 ,G2 , B, Y , Dg, Co, and Cg mean red, green, another green, blue,
luma, orange chroma, and green chroma components, respectively,
the WSST Tog is represented as

Wm =

d−1
X

(3)

wmn + ε (m = 1, 2 and d = 5, 9),(12)|αn (z1 − z2−1 )G1 |,|αn (z1 − z2 )B|,

if G2 prediction,
if R prediction(13)|αn (1 − z1 z2−1 )G1 |,|αn (1 − z1 z2 )B|,

if G2 prediction,
if R prediction

(14)

n=0


w1n =



>

>
Y, Dg , Co , Cg = Tog G1 , G2 , B, R ,

(9)

w2n =

1851

Authorized licensed use limited to: Tsukuba Univ Lib. Downloaded on May 02,2022 at 08:15:30 UTC from IEEE Xplore. Restrictions apply.

Fig. 3. Implementation of the WSSTs.

Table 2. Improved MSEs [%] of Dg components.
5/3 wavelet
9/7 wavelet
Images
WSST→WWSST WSST→WWSST#9-10.12
-8.83
#11
-8.83
-8.17
#34
-11.77-11.54#42-13.55-11.44#45-14.51-14.02#46-10.38-8.97#92-14.53-14.06
#99
-9.35
-9.15
-4.79
-3.97
#103
#107
-14.30-12.81#138-16.81-13.07#482-36.28-33.75#560-15.03-11.98
#784
-8.22
-7.47
-11.32-10.67#845#1145-32.50-30.61#1366-22.90-11.32#4026-16.30-12.95
#4479
-9.57
-9.28
-11.67-10.63#4823
Average-16.00-14.40Fig. 4. A part of Dg components of #482:(top-left) the WSST that uses 5/3 wavelet transforms,(top-right) the WWSST that uses 5/3 wavelet transforms,(bottom-left) the WSST that uses 9/7 wavelet transforms, and(bottom-right) the WWSST that uses 9/7 wavelet transforms.

1,



z

2,



z1 ,



 z1−1 ,z2−1 ,αn =z 1 z2 ,z1−1 z2 ,
z2−1 ,
 z1−1z1 z2−1 ,if n = 0 (center)
if n = 1 (top)
if n = 2 (left)
if n = 3 (right)
if n = 4 (bottom).
if n = 5 (left-top)
if n = 6 (right-top)
if n = 7 (left-bottom)
if n = 8 (right-bottom)(15)

To avoid any divisions by zero in (9)-(11), an extremely small value
ε is added in (12). d means that p pairs of pixels are chosen as
weighted, e.g., in this study, we selected d = 5 in lossless compression and d = 9 in lossy compression experimentally.
4. CFA-SAMPLED RAW IMAGE COMPRESSION
We compared our WWSSTs with the conventional WSSTs [8]. The
5/3 and 9/7 wavelet transforms were used for lossless and lossy compression, respectively. The transformed images were compressedwith JPEG 2000 by using imwrite.m in MATLAB. We used MIT-

Adobe FiveK dataset [14] after converting the dynamic range of theimages from 16 to 14 bits because the sensor data usually had only
about 10 to 14-bit resolution at most (Fig. 5) and subsampling theimages in accordance with the Bayer CFA.
Fig. 4 and Table 2 show a part of Dg components of the several
images transformed by the WSSTs and WWSSTs and their improved
mean squared errors (MSEs)1 [%]. The WWSSTs decorrelated eachcolor component more thoroughly than the WSSTs, especially, they
reduced more energies of Dg components like high frequency information between G1 and G2 components for the CFA-sampled raw
images with many edges. Table 3 shows the lossless bitrates (LBRs)[bpp] in lossless compression and the Bjøntegaard delta (BD) metrics (BD-rates [%] and BD-PSNRs [dB]) between about 0.0625-2
bpp in comparison with the WSSTs in lossy compression. On average compared with the WSSTs, the WWSSTs outperformed byabout 0.01 bpp in LBRs, 0.15 % in BD-rates, and 0.01 dB in BDPSNRs. This fact indicates that the WWSSTs have almost the same
compression efficiency as the WSSTs in this case. Table 4 shows theresults in lossless and lossy compression for the images with manyedges. In this case, the WWSSTs improved the results by about 0.04
bpp in LBRs, 2.10 % in BD-rates, and 0.12 dB in BD-PSNRs com1 Strictly speaking, it is not MSE. However, we express it as MSE because
Dg is like the error between G1 and G2 components.

1852

Authorized licensed use limited to: Tsukuba Univ Lib. Downloaded on May 02,2022 at 08:15:30 UTC from IEEE Xplore. Restrictions apply.

Fig. 5. Images: (first and second rows) general ones: #1, #2, #3, #4, #5, #6, #7, #8, #10, #12, #13, #14, #15, #16, #17, #18, #19, #20, #21,
and #22 and (third and fourth rows) ones with many edges: #9, #11, #34, #42, #45, #46, #92, #99, #103, #107, #138, #482, #560, #784, #845,#1145, #1366, #4026, #4479, and #4823.

Table 3. LBRs [bpp], BD-rates [%], and BD-PSNRs [dB] in losslessand lossy compressions for the general CFA-sampled raw images.
Lossless (LBRs)
Lossy
ImagesWSST WWSST BD-rate BD-PSNR
#1
11.50
11.49
-0.21
0.01
#2
10.90
10.92
1.25
-0.03
#3
8.34
8.33
-0.75
0.04
#4
9.46
9.45
-0.29
0.01
#5
8.03
8.03
-0.23
0.01
9.24
9.24
0.59
-0.02
#6
#7
10.57
10.57
0.40
-0.02
9.32
9.32
0.90
-0.03
#8
#10
8.70
8.68
-0.09
-0.01
#12
9.61
9.60
-0.56
0.03
#13
8.97
8.95
-0.89
0.06
#14
9.97
9.96
-0.65
0.02
#15
8.84
8.82
-0.62
0.03
#16
9.05
9.04
-0.22
0.01
#17
10.86
10.84
-0.66
0.03
9.25
9.24
-0.25
0.01
#18
#19
9.88
9.88
0.05
0.00
#20
9.38
9.36
-0.68
0.03
#21
9.04
9.03
-0.26
0.00
#22
10.39
10.38
-0.05
0.00
Average
9.57
9.56
-0.15
0.01

Table 4. LBRs [bpp], BD-rates [%], and BD-PSNRs [dB] in losslessand lossy compressions for the CFA-sampled raw images with manyedges.
Lossless (LBRs)
Lossy
ImagesWSST WWSST BD-rate BD-PSNR
#9
9.76
9.72
-1.29
0.06
#11
10.90
10.87
-1.18
0.06
#34
10.92
10.88
-1.81
0.09
#42
10.72
10.68
-1.54
0.07
11.30
11.24
-2.05
0.10
#45
#46
9.57
9.54
-1.45
0.08
10.46
10.41
-2.16
0.11
#92
#99
11.09
11.06
-1.30
0.07
#103
9.82
9.80
-1.38
0.07
#107
10.10
10.08
-1.45
0.07
#138
8.42
8.40
-1.37
0.09
#482
10.18
10.08
-6.12
0.36
#560
8.98
8.95
-1.71
0.10
#784
9.69
9.67
-1.68
0.09
10.04
10.01
-1.20
0.06
#845
#1145
9.44
9.38
-4.67
0.27
#1366
9.54
9.51
-2.13
0.13
#4026
10.38
10.32
-4.02
0.28
#4479
9.61
9.60
-1.41
0.07
#4823
10.40
10.37
-2.09
0.11
Average
10.07
10.03
-2.10
0.12

pared with the WSSTs.
for the general CFA-sampled raw images, our WWSSTs could further improve the compression efficiency for the images with manyedges.

5. CONCLUSION
This study introduced the WWSSTs that work especially for the
CFA-sampled raw images with many edges well. The WWSSTs
were obtained by considering that each predict step of the conventional WSSTs is constructed by a combination of two 1-D diagonaltransforms and by weighting them along the edge directions in theimages. As a result, without reducing the compression efficiency

1853

Authorized licensed use limited to: Tsukuba Univ Lib. Downloaded on May 02,2022 at 08:15:30 UTC from IEEE Xplore. Restrictions apply.

6. REFERENCES[1] N.-X. Lian, L. Chang, V. Zagorodnov, and Y.-P. Tan, “Reversing demosaicking and compression in color filter array image
processing: Performance analysis and modeling,” IEEE Trans.
Image Process., vol. 15, no. 11, pp. 3261–3278, Nov. 2006.
[2] K. Chung and Y. Chan, “A lossless compression scheme forbayer color filter array images,” IEEE Trans. Image Process.,
vol. 17, no. 2, pp. 134–144, Feb. 2008.
[3] N. Zhang and X. Wu, “Lossless compression of color mosaic
images,” IEEE Trans. Image Process., vol. 15, no. 6, pp. 1379–
1388, Jun. 2006.
[4] H. S. Malvar and G. J. Sullivan, “Progressive-to-lossless compression of color-filter-array images using macropixel spectralspatial transformation,” in Proc. DCC, Snowbird, UT, Apr.
2012, pp. 3–12.
[5] M. Hernández-Cabronero, M. W. Marcellin, I. Blanes, and
J. Serra-Sagristà, “Lossless compression of color filter arraymosaic images with visualization via JPEG 2000,” IEEE Trans.
Multimedia, vol. 20, no. 2, pp. 257–270, Feb. 2018.
[6] Y. Lee, K. Hirakawa, and T. Q. Nguyen, “Camera-aware multiresolution analysis for raw image sensor data compression,”
IEEE Trans. Image Process., vol. 27, no. 6, pp. 2806–2817,
Jun. 2018.
[7] Y. Lee and K. Hirakawa, “Shift-and-decorrelate lifting: Camra
for lossless intra frame CFA video compression,” IEEE Signal
Prodcess. Lett., vol. 27, no. Jan., pp. 461–465, Jun. 2020.
[8] T. Suzuki, “Wavelet-based spectral spatial transforms for CFAsampled raw camera image compression,” IEEE Trans. Image
Process., vol. 29, no. 1, pp. 433–444, Dec. 2020.
[9] T. Richter, S. Fößel, A. Descampe, and G. Rouvroy, “Bayer
CFA pattern compression with JPEG XS,” IEEE Trans. Image
Process., vol. 30, pp. 6557–6569, Jul. 2021.
[10] W. Ding, F. Wu, X. Wu, and S. Li, “Adaptive directional liftingbased wavelet transform for image coding,” IEEE Trans. Image
Process., vol. 16, no. 2, pp. 416–427, Feb. 2007.
[11] Y. Tanaka, M. Hasegawa, S. Kato, M. Ikehara, and T. Q.
Nguyen., “Adaptive directional wavelet transform based on
directional prefiltering,” IEEE Trans. Image Process., vol. 19,
no. 4, pp. 934–945, Apr. 2010.
[12] T. Yoshida, T. Suzuki, S. Kyochi, and M. Ikehara, “Two dimensional non-separable adaptive directional lifting structureof discrete wavelet transform,” IEICE Trans. Fundamentals.,
vol. E94-A, no. 10, pp. 1920–1927, Oct. 2011.
[13] A. Cohen, I. Daubechies, and J.-C. Feauveau, “Biorthogonalbases of compactly supported wavelets,” Commun. Pure Appl.
Math., vol. 45, no. 5, pp. 485–560, Jun. 1992.
[14] V. Bychkovsky, S. Paris, E. Chan, and F. Durand, “Learning
photographic global tonal adjustment with a database of input/output image pairs,” in Proc. CVPR., Colorado Springs,
CO, USA, Jun. 2011, pp. 97–104.

1854

Authorized licensed use limited to: Tsukuba Univ Lib. Downloaded on May 02,2022 at 08:15:30 UTC from IEEE Xplore. Restrictions apply.