1

Edge-Aware Extended Star-Tetrix Transforms for
CFA-Sampled Raw Camera Image Compression
Taizo Suzuki, Senior Member, IEEE, and Liping Huang, Student Member, IEEE

Abstract—Codecs using spectral-spatial transforms efficiently
compress raw camera images captured with a color filter array(CFA-sampled raw images) by changing their RGB color space
into a decorrelated color space. This study describes two typesof spectral-spatial transform, called extended Star-Tetrix transforms (XSTTs), and their edge-aware versions, called edge-aware
XSTTs (EXSTTs), with no extra bits (side information) and little
extra complexity. They are obtained by (i) extending the StarTetrix transform (STT), which is one of the latest spectral-spatialtransforms, to a new version of our previously proposed waveletbased spectral-spatial transform and a simpler version, (ii)considering that each 2-D predict step of the wavelet transform
is a combination of two 1-D diagonal or horizontal-verticaltransforms, and (iii) weighting the transforms along the edge
directions in the images. Compared with XSTTs, the EXSTTs
can decorrelate CFA-sampled raw images well: they reduce thedifference in energy between the two green components by about
3.38–30.08 % for high-quality camera images and 8.97–14.47 %for mobile phone images. The experiments on JPEG 2000-based
lossless and lossy compression of CFA-sampled raw images show
better performance than conventional methods. For high-quality
camera images, the XSTTs/EXSTTs produce results equal to or
better than the conventional methods: especially for images withmany edges, the type-I EXSTT improves them by about 0.03–0.19
bpp in average lossless bitrate and the XSTTs improve them by
about 0.16–0.96 dB in average Bjøntegaard delta peak signal-tonoise ratio. For mobile phone images, our previous work performthe best, whereas the XSTTs/EXSTTs show similar trends to thecase of high-quality camera images.
Index Terms—Color filter array, edge-aware, raw camera image compression, spectral-spatial transforms, wavelet transforms.

I. I NTRODUCTIONR

AW camera images are mainly created by placing acolor filter array (CFA) between the light sensors andthe camera lens. To economize on hardware, most cameras
capture a color image with a single sensor instead of using
three RGB sensors. In other words, each pixel of such asensor collects a single color component, either red, green,or blue, not all three, and the obtained raw data is called a
CFA-sampled raw image. The Bayer CFA is the most popular
type (Fig. 1). Using a CFA-sampled raw image as is, animage processor performs most of the preprocessing, including
black level correction, white balance, demosaicing [1–4], andgamma correction. In particular, the image quality largely
depends on the performance of the demosaicing processThis work was supported by JSPS Grant-in-Aid for Scientific Research (C)under Grant 22K04084.
T. Suzuki is with the Faculty of Engineering, Information and Systems, University of Tsukuba, Ibaraki 305-8573, Japan (e-mail: taizo@cs.tsukuba.ac.jp).
L. Huang is with Graduate School of Science and Technology, Universityof Tsukuba, Ibaraki 305-8573, Japan (e-mail: huang@wmp.cs.tsukuba.ac.jp).

Fig. 1. Bayer pattern of a particular area of Parrot in the Kodak images
dataset [8] (each 2 × 2 pixel square is a macropixel): (left) RGB full-color
RGB image and (right) simulated CFA-sampled raw image with corresponding
diagram.

that produces the full-color RGB image we see. Moreover,
performing compression after demosaicing (the demosaicingfirst approach) causes redundancy wherein the data volume ofa full-color RGB image is three times that of the original
CFA-sampled raw image, whereas performing compression
before demosaicing (the compression-first approach) can avoid
this redundancy. This compression-first approach also allows
us to process images more freely than the demosaicingfirst one which almost automatically performs various image
processings before compression. In light of these facts, although many traditional image-compression methods take thedemosaicing-first approach, as do standards such as JPEG [5]and JPEG 2000 [6], JPEG XS 2nd Edition [7] takes both
approaches. We believe that the compression-first approach
will get more attention in the future from not only professional
photographers and designers but also typical consumers.
Spectral-spatial transforms for the compression-first approach have been widely researched [9–15]. They change a
CFA-sampled raw image in RGB color space into data in a
decorrelated color space, such as the YDgCbCr or YDgCoCg
color space composed of luma, difference green, and two
chroma components. The spectral redundancy between thedecorrelated components is very small. Since the human visualsystem is not very sensitive to distortion of high-frequency(different green) and chroma components, the strong compression of the decorrelated components will not affect the image
quality much. This study focuses on our previous work on
wavelet-based spectral-spatial transforms (WSSTs) [14] thatare represented by cascading (discrete) wavelet transforms,
such as Haar, 5/3, and 9/7 wavelet transforms, and cover theother spectral-spatial transforms [9–12]. Additionally, there areseveral new spectral-spatial transforms that are not describedin [14]. One is a newer spectral-spatial transform presented
by Lee et al. in [13] for a CFA-sampled raw image coding framework, called camera-aware multi-resolution analysis(CAMRA) [12]. The YDgCoCg2-WSSTs in [14] generalize

2the transform in [13]: the transform in [13] is obtained bysimply applying a 5/3 wavelet transform instead of a Haartransform between the LH and HL subbands to the existingtransforms in [12]. The other one is the Star-Tetrix transform (STT) of the low-latency low-complexity image coding
standard, JPEG XS, developed by Richter et al. [15]. It isconstructed from three 5/3 wavelet transforms. The pixels usedin the predict and update steps are not limited to be within amacropixel; the surrounding pixels are also used to make thepredict results more accurate, as in the case of the existing
WSSTs. On the other hand, for full-color RGB images, (nonredundant) adaptive directional wavelet transforms [16–18],
which adapt the filtering directions along the edge information,
are efficient and popular methods that take into account image
features. Note that, compared with directional transforms, direct use of wavelet transforms for any of the existing spectralspatial transforms amounts to nothing but ignoring the image
features, especially, edge information. However, the existing
directional transforms do so with extra bits (side information),
which adversely affect coding performance, and a significant
amount of complexity.
This study develops extended STT (XSTTs) and edge-aware
XSTTs (EXSTTs) with no side information and little extra
complexity. They are obtained by (i) extending the STT [15]to a new version of the WSSTs [14] and a simpler version,(ii) considering that each 2-D predict step of the wavelet
transforms is a combination of two 1-D diagonal or horizontalvertical transforms, and (iii) weighting the transforms along
the edge directions in the images. Compared with XSTTs, the
EXSTTs can decorrelate CFA-sampled raw images well. In experiments on JPEG 2000-based lossless and lossy compressionof high-quality camera images, our XSTTs/EXSTTs produce
results equal to or better than the conventional methods.
Especially for images with many edges, our XSTTs/EXSTTs
outperform the conventional methods because of their more
efficient decorrelation. Note that in lossy compression, unlike
lossless compression, it is more practical to use XSTTs insteadof EXSTTs because the lossy weights are re-calculated andused in the decoder. In experiments on mobile phone images,
which are relatively noisy, our previous work performs thebest, whereas the XSTTs/EXSTTs show similar trends to thecase of high-quality camera images.
A preliminary version of this study was presented in [19],where we discussed only the edge-aware weighted 2-D diagonal predict steps for the YDgCoCg-WSSTs in [14]. In
addition to that, this paper extends the STTs in [15] to XSTTs
and further presents the edge-aware weighted 2-D horizontalvertical predict steps for the XSTTs.
The remainder of the paper is organized as follows. Section II reviews the conventional methods: WSSTs and STT.
Section III extends the STT to two new versions of the
WSSTs, i.e., the XSTTs, and introduces two types of edgeaware weighted 2-D predict steps to be applied to the XSTTs.
Section IV shows the effect of the edge-aware weighted 2-Dpredict steps and compares the resulting XSTTs/EXSTTs withthe conventional methods in JPEG 2000 for CFA-sampled raw
image compression. Section V concludes the paper.
Notation: Boldface letters represent vectors and matrices. I

TABLE IC OEFFICIENTS OF 5/3 AND 9/7 WAVELET TRANSFORMS .

p0
u0
p1
u1

5/3
−1/2
1/4
0
0

9/7
−1.58613434205992−0.05298011857295
0.882911075530940
0.443506852043967

and O denote a 2 × 2 identity matrix and zero matrix, respectively. Moreover, ·⊤ , ⌊·⌋, and | · | denote the transpose, floor
function (rounding operation), and absolute value, respectively.
Let zi be a horizontal (i = 1) or vertical (i = 2) delay element
and z i = zi−1 . In addition, the size and dynamic range of theimages in the figures have been adjusted for display.
II. R EVIEW AND D EFINITIONS
A. Wavelet-Based Spectral-Spatial Transforms
Our previous work [14] presented WSSTs that cover many
spectral-spatial transforms for CFA-sampled raw image compression. The WSST T is represented as

Y, Dg , C1 , C2

⊤


⊤
= T G1 , G2 , B, R ,(1)

where R, G1 , G2 , B, Y , Dg , C1 , and C2 mean red, green,other green, blue, luma, difference green, chroma, and otherchroma components, respectively. The Ts are classified into
three types: YDgCbCr-WSSTs Tbr , YDgCoCg-WSSTs Tog ,and YDgCoCg2-WSSTs Tog2 , as follows:
W 2 (z 1 , z2 ) O
1
O
P
,
(2)
Tbr = P0OI
O W 3 (z 1 , z2 ) 0




O
W 2 (z 2 ) O
W 2 (z 1 , z2 )Tog = P2P1,OIOW 2 (z 1 , z 2 )
(3)




W 2 (z1 )O
W 2 (z 1 , z2 ) O
P4
Tog2 = P5OIO
W 2 (z1 )
W 2 (z2 )
O
· P3
P ,
(4)
O
W 2 (z2 ) 2
where W 2 (zi ), W 2 (z1 , z2 ), and W 3 (z1 , z2 ) are wavelet
transforms:



0
Y
1
0
1 Uk (zi ),(5)
W 2 (zi ) =Pk (zi ) 101
k=N −1 |{z}|{z}
update step

0
Y


1
W 2 (z1 , z2 ) =0
k=N −1 |predict stepUk (z1 , z2 )1
1
Pk (z1 , z2 )
{z
}|
{z

update step

predict step


0
,
1
}

⊤ 
0
Y
 1
 O
 1 
 1 Uk (z1 )  Pk (z1 )W 3 (z1 , z2 ) =I
Pk (z2 )k=N −1 2 Uk (z2 )
|
{z
}|
{z


update step

predict step

(6)

O
.
I
}
(7)

3

Fig. 2. Implementations of WSSTs (arrows, green-framed macropixels, and red-framed macropixels mean lifting step, G1-to-G2 prediction, and B-to-R
prediction, respectively): (top-to-bottom) YDgCbCr-, YDgCoCg-, and YDgCoCg2-WSSTs.

Pj (j ∈ N) is a 4 × 4 permutation matrix,



0 0
0 1 0 0
1 0
1 0 0 0
P0 = 
0 0 1 0 , P1 = 0 1
0 0
0 0 0 1
0 0
1 0 0 0
1 0
0 0 1 0
P2 = 
0 0 0 1 , P3 = 0 0
0 1
0 1 0 0

0 0
0 1 0 0
0 1
0 0 1 0, P5 = P4 = 
1 0
1 0 0 00 0
0 0 0 1

1
0
0
0
1
0
0
0
1
0
0
0


0
0
,
0
1

0
0
,
1
0

0
0
,
0
1

with JPEG XS. It consists of four steps based on 5/3 wavelet
transforms. The first step generates chroma components Cb
and Cr by predicting R and B from the four surrounding
green components Gx (x = {l, r, t, b}), where l, r, t, andb respectively indicate the sample position to the left, right,
top, and bottom of the current pixel, as l
G + Gr + Gt + Gb,
(13)
Cb = B −4
 l
G + Gr + Gt + Gb
Cr = R −
.
(14)
4
(8)

and Pk (zi ), Uk (zi ), Pk (z1 , z2 ), and Uk (z1 , z2 ) are polynomials with coefficients pk and uk :Pk (zi ) = (1 + z i )pk ,(9)

Uk (zi ) = (1 + zi )uk ,(10)
1
Pk (z1 , z2 ) = (1 + z 1 + z 2 + z 1 z 2 )pk ,(11)2
1
Uk (z1 , z2 ) = (1 + z1 + z2 + z1 z2 )uk .
(12)
2
Table I shows the coefficients pk and uk in the 5/3 wavelet
transforms (N = 1) and 9/7 wavelet transforms (N = 2), andFig. 2 shows implementations of the existing WSSTs. The
pixels used in the predict and update steps are not limited to
be within a macropixel; the surrounding pixels are also used
to make the predict results more accurate. However, because
they do not consider image features, their predictions may not
be so accurate in some cases.
B. Star-Tetrix Transform
The STT, presented by Richter et al. [15], is a spectralspatial transform for CFA-sampled raw image compression

The second step generates the luma components Y1 and Y2 by
updating G from the four surrounding Cbx s and Crx s as lCr + Crr + Cbt + Cbb
Y1 = G +
,
(15)
8
 t

Cr + Crb + Cbl + Cbr
Y2 = G +.
(16)
8
For simplicity, the white-balancing constants defined in [15]will not be considered in this study. The third step generates
the luma difference component ∆ by predicting Y1 from thefour surrounding Y2 s, as%$
Y2l,t + Y2r,t + Y2l,b + Y2r,b.
(17)∆ = Y1 −4
The last step generates the final luma component Y by
updating Y2 from the four surrounding ∆s, as l,t∆ + ∆r,t + ∆l,b + ∆r,b
Y = Y2 +.
(18)
8Although the STT here looks different from the WSSTs, we
can extend (generalize) the STT to the WSSTs as describedin the next section.

4III. E XTENDED S TAR -T ETRIX T RANSFORMS AND
E DGE -AWARE W EIGHTED 2-D P REDICT S TEPS

B. Type-II Extended Star-Tetrix Transforms: A Simpler Version
than Type-I Extended Star-Tetrix Transforms

A. Type-I Extended Star-Tetrix Transforms: A New Waveletbased Spectral-Spatial Transform

Definition: Here, we describe another type of XSTT; called
the “type-II XSTTs” or “XSTT-IIs.” These are simpler thanthe XSTT-Is and a fifth version of the WSSTs (third versionof the YDgCbCr-WSSTs). The XSTT-II SII is represented asfollows:

Definition: Although the original STT [15] consists of three
5/3 wavelet transforms as described above, we can extend
it to a fourth version of the WSSTs (second version of
the YDgCbCr-WSSTs). Since another type of extended STT(XSTT) will be introduced in the next subsection, we willrefer to the XSTTs in this subsection as “type-I XSTTs” or“XSTT-Is” to distinguish between them. The XSTT-I SI isrepresented as follows:
SI =

 0 



I Uk
I O
W 2 (z 1 , z2 ) O Y, (19)OI
O I
Pk I
k=N −1 |{z } | {z }update step

predict step

where






1 Pk (z2 ) Pk (z1 )1 Uk (z2 ) Uk (z 1 )Pk =, Uk =.
2 Pk (z 1 ) Pk (z 2 )2 Uk (z1 ) Uk (z 2 )(20)
Although we have omitted the white-balancing constants
defined in [15], the parameters can be simply applied toW 2 (z 1 , z2 ) in (19) if we desire it. The top of Fig. 4 shows animplementation of the XSTT-Is. The XSTT-Is will be further
extended to edge-aware transforms later.
Remark-1: In the case of the 5/3 wavelet transforms, thefirst, second, and third steps in (19) represent the same process
as (13)-(14), (15)-(16), and (17)-(18), respectively; i.e., the
XSTT-I that uses 5/3 wavelet transforms is identical to theoriginal STT. In addition, Dg in the XSTT-I is denoted as ∆in the original STT.
Remark-2: If there are no rounding operations in the lifting
steps, we find that the XSTT-I that uses Haar transforms is
identical to the YDgCbCr-WSST that uses Haar transforms:

1/4
1/4 1/4 1/4
 −1
1
0
0 

SI = 
−1/2 −1/2
1
0 
−1/2 −1/2
0
1



1 1/2 0 0
1 0 0 0
0 1 0 0 −1 1 0 0=
0 0 1 0  0 0 1 0
0 0 0 10 0 0 1

1 0 1/4 1/41
0
0 0
0 1 1/4 1/4  0
1
0 0

 . (21)
·
0 0


1
0
−1/2 −1/2 1 0
0 00
1
−1/2 −1/2 0 1
The implementation is shown at the top of Fig. 3. Consequently, the XSTT-Is can be considered to be a second versionof the YDgCbCr-WSSTs. Note that the XSTT-I that uses
Haar transforms outputs slightly different results from the
YDgCbCr-WSST that uses Haar transforms due to the rounding operations on the lifting steps in the actual implementation.



e 0 W 2 (z 1 , z2 ) O
I O
I U
,
SII =
O
I P0 IO I| {z }| {z }


update step

(22)

predict step

where


1 U0 (z2 ) U0 (z 1 )
e
U0 =
.
0
0
2

(23)

The XSTT-IIs are considered to be special cases of WSSTs
because wavelet transforms with N ≥ 2, such as 9/7 wavelet
transforms, can only be applied to the second step in (22). The
bottom of Fig. 4 shows an implementation of the XSTT-IIs. As
with the XSTT-Is, they will be further extended to edge-awaretransforms later.
Derivation: The XSTT-II in (22) is obtained with thefollowing simple procedure. First, the XSTT-I that uses Haartransforms in (21) is rewritten as
1/4
1/4 1/4 1/4
 −1
1
0
0 

SII = −1/2 −1/2 1
0 −1/2 −1/2 01

1 1/2 0 0
1 0 1/4 1/40 1 00  0 1 0 0=
0 0 10  0 0 1 0
0 0 0 1
0 0 0
1



1
0
0 0
1 0 0 0
−1 1 0 0  0
1
0 0


·
 0 0 1 0 −1/2 −1/2 1 0 . (24)0 0 0 1 −1/2 −1/2 0 1
Let SII be the XSTT-II that uses Haar transforms. Next,
in accordance with the extension method described in [14],the XSTT-II that uses Haar transforms in (24) can easily be
extended to WSSTs in the form of (22).
Remark-1: The XSTT-IIs are simpler than the XSTT-Is bee 0 in (23) does not require U0 (z1 ) or U0 (z 2 ) belonging
cause Uto the bottom row of U 0 in (20), i.e., U0 (z1 ) = U0 (z 2 ) = 0.
Remark-2: Like the XSTT-I, if there are no rounding
operations in the lifting steps, the XSTT-II that uses Haartransforms is identical to the YDgCbCr-WSST that uses Haartransforms, as shown in (24). The implementation is shownat the bottom of Fig. 3. Consequently, the XSTT-IIs can beconsidered to be a third version of the YDgCbCr-WSSTs. Notethat the XSTT-II that uses Haar transforms outputs slightly
different results from the XSTT-I and YDgCbCr-WSST thatuse Haar transforms due to the rounding operations.

5Fig. 3. Implementations of XSTTs that use Haar transforms (arrows, green-framed macropixels, and magenta-framed macropixels mean lifting step, G1-to-G2
prediction, and (G1 , G2 )-to-R and (G1 , G2 )-to-B predictions, respectively): (top) XSTT-Is and (bottom) XSTT-IIs.

Fig. 4. Implementations of XSTTs (arrows, green-framed macropixels, and magenta-framed macropixels mean lifting step, G1-to-G2 prediction, and(G1 , G2 )-to-R and (G1 , G2 )-to-B predictions, respectively): (top) XSTT-Is and (bottom) XSTT-IIs.

Fig. 5. Weights for the 2-D diagonal predict step (dashed arrows mean to
calculate the difference between two pixels): (top) for G1 -to-G2 predictionand (bottom) B-to-R prediction.

Fig. 6. Relation between weight calculation and edge direction detection(dashed arrows mean the difference between two pixels and the red areas
are the detectable edge direction ranges): (top-left) left diagonal weight, (topright) right diagonal weight, (bottom-left) vertical weight, and (bottom-right)
horizontal weight.

C. Edge-Aware Weighted 2-D Diagonal Predict StepsIn this subsection, we extend the polynomial Pk (z1 , z2 ) in(11), which is in the 2-D diagonal predict step of the wavelet
transform W 2 (z1 , z2 ) in (6), toW1 (1 + z 1 z 2 ) + W2 (z 1 + z 2 )pk ,Pek (z1 , z2 ) =W1 + W2
where Wm (m = 1, 2) is a weight,XWm =wmn + ε,

(25)

(26)

n

wmn is automatically determined in accordance with the image
features as described later, and an extremely small value ε isadded to (26) to avoid division by zero. Hence, the original
2-D diagonal predict step is divided into two 1-D diagonal(left- and right-diagonal) predict steps and they are weighted
to consider the image features. When W1 = W2 = 1, it is clearthat Pek (z1 , z2 ) = Pk (z1 , z2 ). Consequently, for the cases of
G1 -to-G2 and B-to-R predictions, we can rewrite Pek (z1 , z2 )6in (25), asW1 (1 + z1 z 2 ) + W2 (z1 + z 2 )Pek (z 1 , z2 ) =pk
W1 + W2
for G1 -to-G2 prediction, (27)
W1 (1 + z1 z2 ) + W2 (z1 + z2 )pk
Pek (z 1 , z 2 ) =W1 + W2
for B-to-R prediction, (28)and define w1n and w2n as(|Zn (z1 − z 2 )G1 |γ
w1n =|Zn (z1 − z2 )B|γ(
w2n =|Zn (1 − z1 z 2 )G1 |γ|Zn (1 − z1 z2 )B|γ

for G1 -to-G2 prediction,
for B-to-R prediction(29)
for G1 -to-G2 prediction,
for B-to-R prediction(30)

where γ ∈ R is an arbitrary adjustment parameter that
evaluates the edge-likeness (Pek (z1 , z2 ) = Pk (z1 , z2 ) if γ = 0)
and Zn is a delay:1
if n = 0 (center)





z2
if n = 1 (top)





z1
if n = 2 (left)





z
if n = 3 (right)
 1
Zn = z 2.
(31)
if n = 4 (bottom)

z1 z2 if n = 5 (top-left)





z 1 z2 if n = 6 (top-right)





z1 z 2 if n = 7 (bottom-left)
z 1 z 2 if n = 8 (bottom-right)
Figure 5 overviews the weights for the edge-aware weighted
2-D diagonal predict steps. γ was empirically set to γ = 1
in the lossless compression experiment and γ = 1/2 in the
lossy compression experiment. Since the edge-aware weighted
2-D diagonal predict steps can be applied to any Pk (z1 , z2 )in (6), we can extend all of the WSSTs and XSTTs toefficiently decorrelate the CFA-sampled raw images while
taking the edge information into account. Also, to avoid extra
bits and a significant amount of complexity, the weights are
not transmitted to the decoder and are re-calculated in the
decoder. Note that, as in Section IV-C, the weights between
the encoder and decoder tend to be more different at higher
lossy compression levels.
D. Edge-Aware Weighted 2-D Horizontal-Vertical Predict
Steps
Like the edge-aware weighted 2-D diagonal predict steps in(25), we can also extend the polynomial Pk (zi ) in (9), which isin the 2-D horizontal-vertical predict step in (23), as follows:Wi(1 + z i )pk .
(32)
Pek (zi ) =W1 + W2
When W1 = W2 = 1, it is clear that Pek (zi ) = Pk (zi ).
Naturally, we can rewrite Pek (zi ) in (32) asWi
Pek (z i ) =(1 + zi )pk .
(33)
W1 + W2

Fig. 7. Weights for the 2-D horizontal-vertical predict steps (dashed arrows
mean to calculate the difference between two pixels): (top) (G1 , G2 )-to-R
prediction and (bottom) (G1 , G2 )-to-B prediction.

Moreover, we can rewrite wmn as(|Zn (1 − z2 )G2 |γ for (G1 , G2 )-to-R prediction
w1n =,|Zn (1 − z 2 )G1 |γ for (G1 , G2 )-to-B prediction(34)(
γ|Zn (1 − z1 )G1 |for (G1 , G2 )-to-R prediction
w2n =
,
γ
|Zn (1 − z 1 )G2 |for (G1 , G2 )-to-B prediction(35)
where γ and Zn are the same as in the case of the edge-aware
weighted 2-D diagonal predict steps. Figure 7 overviews theweights for the 2-D horizontal-vertical predict steps. Since
the vertical (horizontal) weight wmn detects the horizontal(vertical) edge direction with a 90◦ range, the top and bottom(left and right) weights are omitted. Like the edge-aware
weighted 2-D diagonal predict steps, γ was empirically set
as γ = 1 in the lossless compression experiment and γ = 1/2
in the lossy compression experiment. Since the edge-aware
weighted 2-D horizontal-vertical predict steps can be applied
to only Pk (zi ) in (20), unlike the diagonal case, and Pk (zi ) in(20) is included only in the XSTTs, we will be able to extend
only the XSTTs. Also, as in the diagonal case, the weights are
re-calculated in the decoder.
IV. E XPERIMENTAL R ESULTS
A. Preparation
We compared our XSTTs and EXSTTs, which are the
XSTTs that use the edge-aware weighted 2-D predict steps,
with existing spectral-spatial transforms including simple ones
which do not apply white balance or gamma correction tothemselves for CAMRA [12], the WSSTs [14], the STT [15]which do not apply white-balancing constants to themselves,
and the weighted WSSTs (WWSSTs) [19].1 For reference, we
1 The YDgCoCg-WSSTs/WWSSTs performed the best among the three
types of WSSTs/WWSSTs in preliminary experiments, so the comparisons presented here show only YDgCoCg-WSSTs/WWSSTs; hereafter, the YDgCoCg-WSSTs/WWSSTs will simply be referred to as“WSSTs/WWSSTs.” In addition, the weights of WWSST were updated tothe ones in this study.

7Fig. 8. Test images developed from raw data by a simple camera processing pipeline: (top) general images (#0500, #1000, #1500, #2000, #2500, #3000,#3500, #4000, #4500, and #5000), (middle) images with many edges (#0092, #0482, #0548, #1137, #1145, #1463, #2239, #2912, #3394, and #3419), and(bottom) mobile phone images (m01, m02, m03, m04, m05, m06, m07, m08, m09, and m10).

compared them with direct JPEG 2000 compression of fullcolor RGB images obtained by a simple camera processing
pipeline (black level correction, white balance, demosaicing [1], and gamma correction), as shown in [20]. The 5/3
and 9/7 wavelet transforms2 were used in the lossless andlossy compressions based on the spectral-spatial transforms;
the XSTT-II and EXSTT-II have 5/3 wavelet transforms forthe first and last steps even for lossy compression because of
their special structures. Note that the XSTT-I that uses 5/3
wavelet transforms is identical to the STT and the one thatuses 9/7 wavelet transforms can be regarded as an extended
version of the STT, as indicated above. The transformed
images were adjusted to positive values and compressed with
JPEG 2000 [6]3 by using ‘imwrite.m’ with the default settings(except for compression ratio) in MATLAB. To investigate
the effect on high-quality camera images, we used 20 DNG
data, whose bit depths are 12 or 14 bits, in the MIT-Adobe
FiveK dataset [21] after loading them via ‘rawread.m’ in
MATLAB and forming them into an “RGGB” array. The highquality camera images included ten general images used in[14] and ten intuitively selected images that seemed to have
many edges. To investigate the effect on mobile phone images,
which are noisier than the images in the MIT-Adobe FiveK
dataset, we used ten DNG data, whose bit depths are 12 bits,
acquired with an iPhone SE (3rd generation). Figure 8 shows
the 30 test images. In addition, we used the lossless bitrates(LBRs) [bpp] in lossless compression and the Bjøntegaard
delta peak signal-to-noise ratios (BD-PSNRs) [dB] between
about 2–5 bpp in lossy compression for a fair comparison. We
selected the macropixel spectral-spatial transform (MSST) [10]as the basis for the comparison using the BD-PSNRs andcomputed the BD-PSNRs after the CFA-sampled images weredeveloped into full-color RGB images by using a simple
camera processing pipeline similar to that used for direct JPEG
2000 compression.
B. Decorrelation of CFA-Sampled Raw Images
Figure 9 shows only the decorrelated (transformed) subband
images with the XSTT-I and EXSTT-I that use 9/7 wavelet
2 We did not apply rounding operations to the 9/7 wavelet transforms forbetter lossy compression.
3 JPEG 2000 that uses the wavelet transforms has high affinity to any
spectral-spatial transform. Of course, other codecs, such as JPEG XR and
JPEG XS, can be applied to any of the spectral-spatial transforms.

Fig. 9. Decorrelated subband images of #0482 (clockwise from the top-left:
Y , Dg , C2 , and C1 ): (left) XSTT-I and (right) EXSTT-I that use 9/7 wavelet
transforms.

1000

0

-1000

Fig. 10. Part of Dg components of #0482 represented in pseudo color: (top)
XSTT-I and EXSTT-I that use 5/3 wavelet transforms, (bottom) XSTT-I and
EXSTT-I that use 9/7 wavelet transforms.

transforms because the differences depending on the transforms were not clear. The grayscale-transformed images weredirectly compressed with JPEG 2000.
To clearly show the effect of the edge-aware weighted
2-D predict steps, Fig. 10 and Table II show parts of the
Dg components, the high-frequency information between theG1 and G2 components, of the images transformed by the
XSTTs/EXSTTs and the reduction (improvement) in meansquared error (MSE)4 [%] of the Dg components after applying the edge-aware weighted 2-D predict steps to the
XSTTs. Since the Dg components in the XSTT-II/EXSTTIIs look almost identical to the ones in the XSTT-I/EXSTTIs, we omitted the XSTT-II/EXSTT-IIs from Fig. 10. The
4 Strictly speaking, these values are not MSEs. However, we indicate themas such because Dg is like the error between the G1 and G2 components.

8

TABLE III MPROVEMENT IN MSE S [%] OF DG COMPONENTS AFTER INCORPORATING EDGE - AWARE WEIGHTED 2-D PREDICT STEPS IN THE XSTT S( LOWER MSE S ARE BETTER ).

#0500
#1000
#1500
#2000
#2500
#3000
#3500
#4000
#4500
#5000
Avg.

type-I
type-II
5/3
9/7
5/3
9/7
+1.42 +1.51 +1.19 +1.37+1.24 +1.40 +0.36 +1.27−17.00 −10.91 −13.59 −9.29−13.17 −8.86 −9.03 −5.49+0.66 +0.68 +0.41 +0.73−4.89 −2.78 −4.29 −1.99−16.09 −7.09 −14.83 −5.91−8.22 −8.53 −1.82 −1.17−13.33 −9.25 −11.80 −7.82−6.67 −3.32 −6.59 −3.25−9.84 −7.66 −5.92 −3.38

#0092
#0482
#0548
#1137
#1145
#1463
#2239
#2912
#3394
#3419
Avg.

type-I
5/3
9/7−16.17 −12.38−41.66 −34.05−17.04 −10.95−31.92 −23.62−26.42 −19.31−17.91 −14.20−26.81 −19.08−25.99 −12.83−25.95 −20.36−41.44 −38.39−30.08 −24.82

type-II
5/3
9/7−14.72 −11.48−39.25 −31.76−14.85 −9.83−30.20 −22.74−26.14 −18.68−17.35 −14.52−25.11 −18.24−24.51 −10.31−23.04 −18.22−38.31 −36.41−27.81 −23.23

m01
m02
m03
m04
m05
m06
m07
m08
m09
m10
Avg.

type-I
5/3
9/7−9.21 −5.71−6.66 −3.64−20.58 −16.22−17.42 −11.34−29.04 −23.56−11.40 −7.72−10.03 −6.22−7.87 −3.79−22.69 −18.72−9.25 −4.89−14.47 −10.26

type-II
5/3
9/7−8.43 −5.01−6.18 −3.17−15.78 −12.72−16.59 −10.83−24.50 −19.60−10.24 −6.94−9.34 −5.72−8.13 −4.16−19.96 −16.67−9.22 −5.07−12.81 −8.97

1

0.5

0

Fig. 11. Weights for G1-to-G2 predictions, which generate Dg components, in the EXSTT-Is for part of #0482 represented in pseudo color: (top) W1 /(W1 +W2 ) in 5/3 wavelet transforms, (bottom) the first W1 /(W1 + W2 ) in 9/7 wavelet transforms, (first column) ideal weights on the encoder, and (second-to-lastcolumns) lossy weights on the decoder when 5, 4, 3, and 2 bpp.

EXSTTs reduced the average MSEs of the Dg components
by about 3.38–9.84 % for general images, 23.23–30.08 % forimages with many edges, and 8.97–14.47 % for mobile phone
images. Figure 11 shows the corresponding weights for thepart in Fig. 10. We omitted the second weights in the 9/7
wavelet transforms, which looked almost identical to the firstweights. Figures 10 and 11 show that the weights assigned
along the edge directions helped to reduce the Dg energies.
There were also slight differences in weight between the 5/3
and 9/7 wavelet transforms. On the other hand, unfortunately,
the weights between the encoder and decoder tended to be
more different for higher lossy compression levels. Since thedifferences may affect coding performance, we will investigate
this finding in Section IV-C.
C. CFA-Sampled Raw Image Compression
1) Effect of Edge-Aware Weighted 2-D Predict Steps:
Tables III and IV show the improvements in the losslessand lossy compressions for the CFA-sampled raw images
after incorporating the edge-aware weighted 2-D predict steps
in the XSTTs: the differences in LBR and BD-PSNR wereobtained by subtracting the LBR and BD-PSNR of the originaltransforms from those of the edge-aware transforms in losslessand lossy compression, respectively. For lossless compression,
although all EXSTTs hardly showed any differences on thegeneral and mobile phone images, they gave the best resultsin the case of images with many edges. In contrast, forlossy compression, the EXSTTs gave worse results than the
XSTTs because we re-calculated lossy weights on the decoder(Fig. 11). If we would like to use the ideal weights on the
decoder, the encoder-calculated weights must be transmitted
to the decoder together with a lot of side information and it ismore practical to use the XSTTs instead of the EXSTTs forlossy compression.
2) Comparisons with Other Methods: Table V, Table VI,
and Fig. 12 show the LBRs in lossless compression, BDPSNRs in lossy compression, and rate-distortion (R-D) curves
in lossy compression, respectively. For lossless compression,
the compression-first approaches using the spectral-spatialtransforms clearly outperformed the direct JPEG 2000 compression of full-color RGB images, because the full-color RGB
images had three times the components compared with a CFAsampled raw image. For lossy compression, the compressionfirst approaches using the spectral-spatial transforms outperformed the direct JPEG 2000 compression at higher bitrates(4 bits or more), but not at lower bitrates. For high-quality

9

TABLE IIII MPROVEMENT IN LBR S [ BPP ] FOR LOSSLESS COMPRESSION AFTER INCORPORATING EDGE - AWARE WEIGHTED 2-D PREDICT STEPS IN THE XSTT S( LOWER LBR S ARE BETTER ).

#0500
#1000
#1500
#2000
#2500
#3000
#3500
#4000
#4500
#5000
Avg.

type-I
+0.01
+0.01
±0.00
+0.01
+0.01
±0.00
±0.00
+0.01
±0.00
±0.00
±0.00

type-II
+0.01
+0.01
±0.00
+0.01
+0.01
±0.00
±0.00
+0.01
±0.00
±0.00
±0.00

#0092
#0482
#0548
#1137
#1145
#1463
#2239
#2912
#3394
#3419
Avg.

type-I
−0.02
−0.08
−0.02
−0.02
−0.02
−0.02
−0.02
−0.04
−0.02
−0.02
−0.03

type-II
−0.02
−0.08
−0.02
−0.02
−0.02
−0.02
−0.02
−0.04
−0.02
−0.02
−0.03

m01
m02
m03
m04
m05
m06
m07
m08
m09
m10
Avg.

type-I
±0.00
±0.00
−0.01
−0.01
−0.01
−0.01
−0.01
−0.01
−0.01
−0.01
−0.01

type-II
±0.00
±0.00
−0.01
−0.01
−0.01
−0.01
−0.01
−0.01
−0.01
−0.01
−0.01

TABLE IVI MPROVEMENT IN BD-PSNR S [ D B] FOR LOSSY COMPRESSION AFTER INCORPORATING EDGE - AWARE WEIGHTED 2-D PREDICT STEPS IN THE XSTT S( HIGHER BD-PSNR S ARE BETTER , AND THE VALUES IN () INDICATE THE IDEAL WEIGHTS WITHOUT SIDE INFORMATION CONSIDERED ).

#0500
#1000
#1500
#2000
#2500
#3000
#3500
#4000
#4500
#5000
Avg.

type-I−0.27 (−0.06)−0.36 (−0.11)−0.42 (−0.03)−0.30 (−0.08)−0.33 (−0.11)−0.82 (−0.04)−0.52 (−0.11)−0.78 (−0.10)−0.15 (−0.02)−0.16 (−0.04)−0.41 (−0.07)

type-II−0.15 (−0.07)−0.15 (−0.05)−0.78 (−0.03)−0.09 (−0.01)−0.15 (−0.06)−0.64 (+0.01)−0.10 (−0.05)−0.23 (−0.04)−0.06 (±0.00)−0.07 (−0.02)−0.24 (−0.03)

#0092
#0482
#0548
#1137
#1145
#1463
#2239
#2912
#3394
#3419
Avg.

type-I−0.62 (+0.10)−0.24 (+0.19)−0.06 (±0.00)−0.08 (+0.01)−0.06 (+0.03)−0.21 (+0.07)−0.03 (+0.05)−0.03 (+0.08)−2.13 (+0.03)−0.67 (−0.01)−0.41 (+0.06)

camera images, the XSTTs/EXSTTs produced results equal
to or better than the conventional spectral-spatial transforms:
especially for images with many edges, the EXSTT-I improved them by about 0.03–0.19 bpp in average LBR and
the XSTTs improved them by about 0.16–0.96 dB in average
BD-PSNR. In addition, for mobile phone images, i.e. noisy
images, the WSSTs/WWSSTs performed the best, whereas
the XSTTs/EXSTTs showed similar tends to the case of highquality camera images. Also, as shown in Fig. 13, there wasno noticeable problem with the images even when they had
been compressed with 2 bpp and developed with the simple
camera processing pipeline.
V. C ONCLUSION
This study described two types of XSTT and their edgeaware versions, EXSTTs, with no side information and little
extra complexity for CFA-sampled raw image compression.
They were obtained by (i) extending the STT, which is one of
the latest spectral-spatial transforms, to a new version of ourpreviously proposed WSSTs and a simpler version, (ii) considering that each 2-D predict step of the wavelet transforms
is a combination of two 1-D diagonal or horizontal-verticaltransforms, and (iii) weighting the transforms along the edge
directions in the images. Compared with XSTTs, the EXSTTs
decorrelated the images well. For images with many edges, our
XSTTs/EXSTTs produced results better than the conventional
methods in lossless and lossy compressions without reducing
the compression efficiency for general images. For mobile
phone images, our previous work performed the best, whereas
the XSTTs/EXSTTs showed similar trends to the case of highquality camera images.

type-II−1.54 (+0.05)−0.08 (+0.25)±0.00 (+0.02)+0.01 (+0.05)+0.01 (+0.07)−0.12 (+0.09)−0.04 (+0.03)+0.04 (+0.09)−0.51 (+0.04)−0.09 (+0.08)−0.23 (+0.08)

m01
m02
m03
m04
m05
m06
m07
m08
m09
m10
Avg.

type-I−0.29 (−0.04)−0.11 (±0.00)−1.59 (−0.15)−0.15 (−0.05)−0.10 (±0.00)−0.13 (+0.05)−0.07 (±0.00−0.09 (−0.01)−0.25 (±0.00−0.04 (+0.04)−0.28 (−0.01)

type-II−0.09 (−0.04)−0.07 (−0.02)−1.39 (−0.01)−2.82 (+0.01)−0.41 (+0.01)−1.38 (+0.01)−0.07 (−0.01)−0.06 (−0.03)−1.16 (−0.02)+0.01 (+0.05)−0.74 (−0.01)

ACKNOWLEDGMENT
The authors would like to thank the anonymous reviewers
and Dr. K. Shirai for providing many constructive suggestions
that significantly improved the presentation of this paper.
R EFERENCES[1] H. S. Malvar, L. He, and R. Cutler, “High quality linear interpolation
for demosaicing of Bayer-patterned color images,” in Proc. ICASSP,
Montreal, Quebec, Canada, May 2004, pp. 485–488.
[2] D. Kiku, Y. Monno, M. Tanaka, and M. Okutomi, “Beyond color
difference: Residual interpolation for color image demosaicking,” IEEE
Trans. Image Process., vol. 25, no. 3, pp. 1288–1300, Mar. 2016.
[3] D. S. Tan, W.-Y. Chen, and K.-L. Hua, “DeepDemosaicking: Adaptive
image demosaicking via multiple deep fully convolutional networks,”
IEEE Trans. Circuits Syst. Video Technol., vol. 27, no. 5, pp. 2408–
2419, May 2018.
[4] Y. Wang, S. Yin, S. Zhu, Z. Ma, R. Xiong, and B. Zeng, “NTSDCN: New
three-stage deep convolutional image demosaicking network,” IEEE
Trans. Circuits Syst. Video Technol., vol. 31, no. 3, pp. 3725–3729, Nov.
2020.
[5] G. K. Wallace, “The JPEG still picture compression standard,” IEEE
Trans. Consum. Electr., vol. 38, no. 1, pp. xviii–xxxiv, Feb. 1992.
[6] A. Skodras, C. Christopoulis, and T. Ebrahimi, “The JPEG2000 still
image compression standard,” IEEE Signal Process. Mag., vol. 18, no.
5, pp. 36–58, Sep. 2001.
[7] A. Descampe, T. Richter, T. Ebrahimi, S. Fößel, J. Keinert, T. Bruylants,
P. Pellegrin, C. Buysschaert, and G. Rouvroy, “JPEG XS–A new
standard for visually lossless low-latency lightweight image coding,”
Proc. IEEE, vol. 109, no. 9, pp. 1559–1577, May 2021.
[8] “Kodak Lossless True Color Image Suite,” Kodak [Online], Available:
http://r0k.us/graphics/kodak/.
[9] N. Zhang and X. Wu, “Lossless compression of color mosaic images,”
IEEE Trans. Image Process., vol. 15, no. 6, pp. 1379–1388, Jun. 2006.
[10] H. S. Malvar and G. J. Sullivan, “Progressive-to-lossless compressionof color-filter-array images using macropixel spectral-spatial transformation,” in Proc. DCC, Snowbird, UT, Apr. 2012, pp. 3–12.

10

TABLE VLBR S [ BPP ] IN LOSSLESS COMPRESSION .

#0500
#1000
#1500
#2000
#2500
#3000
#3500
#4000
#4500
#5000
Avg.
#0092
#0482
#0548
#1137
#1145
#1463
#2239
#2912
#3394
#3419
Avg.
m01
m02
m03
m04
m05
m06
m07
m08
m09
m10
Avg.

Direct(37.56)(34.84)(33.39)(34.14)(36.24)(37.94)(29.80)(39.72)(34.41)(35.30)(35.33)(34.03)(34.17)(35.69)(34.06)(34.02)(37.46)(33.99)(34.04)(36.00)(32.11)(34.56)(36.72)(35.87)(30.83)(35.27)(34.61)(31.71)(36.96)(36.98)(31.36)(36.23)(34.65)MSST [10]
6.43
6.38
8.44
5.27
6.27
5.98
7.46
8.62
7.12
6.14
6.81
7.21
9.19
6.98
8.31
8.96
7.46
6.30
6.16
8.50
8.29
7.73
7.56
7.44
6.61
7.10
6.23
7.13
6.45
8.35
6.58
7.75
7.12

CAMRA [12]
6.41
6.40
8.38
5.25
6.31
5.97
7.42
8.66
7.04
6.11
6.80
7.15
9.07
6.81
8.22
8.93
7.58
6.23
6.06
8.45
8.19
7.67
7.49
7.36
6.69
6.99
6.21
7.28
6.38
8.21
6.74
7.64
7.10

WSST [14]
6.39
6.37
8.32
5.23
6.25
5.92
7.36
8.62
7.02
5.98
6.75
7.05
9.05
6.84
8.22
8.85
7.53
6.21
5.99
8.40
8.15
7.63
7.44
7.32
6.50
6.96
6.16
6.92
6.30
8.14
6.44
7.57
6.98

STT [15]
6.40
6.40
8.32
5.19
6.28
5.91
7.35
8.65
6.94
6.01
6.75
6.88
9.01
6.77
8.17
8.83
7.48
6.15
5.96
8.37
8.08
7.57
7.41
7.27
6.53
6.94
6.19
7.00
6.30
8.09
6.57
7.55
6.99

[11] M. Hernández-Cabronero, M. W. Marcellin, I. Blanes, and J. SerraSagristà, “Lossless compression of color filter array mosaic images
with visualization via JPEG 2000,” IEEE Trans. Multimedia, vol. 20,
no. 2, pp. 257–270, Feb. 2018.
[12] Y. Lee, K. Hirakawa, and T. Q. Nguyen, “Camera-aware multi-resolution
analysis for raw image sensor data compression,” IEEE Trans. Image
Process., vol. 27, no. 6, pp. 2806–2817, Jun. 2018.
[13] Y. Lee and K. Hirakawa, “Shift-and-decorrelate lifting: CAMRA for
lossless intra frame CFA video compression,” IEEE Signal Process.
Lett., vol. 27, pp. 461–465, Jan. 2020.
[14] T. Suzuki, “Wavelet-based spectral-spatial transforms for CFA-sampled
raw camera image compression,” IEEE Trans. Image Process., vol. 29,
no. 1, pp. 433–444, Dec. 2020.
[15] T. Richter, S. Fößel, A. Descampe, and G. Rouvroy, “Bayer patterncompression with JPEG XS,” IEEE Trans. Image Process., vol. 30, pp.
6557–6569, Jul. 2021.
[16] W. Ding, F. Wu, X. Wu, and S. Li, “Adaptive directional lifting-based
wavelet transform for image coding,” IEEE Trans. Image Process., vol.
16, no. 2, pp. 416–427, Feb. 2007.
[17] Y. Tanaka, M. Hasegawa, S. Kato, and T. Q. Nguyen, “Adaptive
directional wavelet transform based on directional prefiltering,” IEEE
Trans. Image Process., vol. 19, no. 4, pp. 934–945, Apr. 2010.
[18] T. Yoshida, T. Suzuki, S. Kyochi, and M. Ikehara, “Two dimensional
non-separable adaptive lifting structure of discrete wavelet transform,”
IEICE Trans. Fundamentals, vol. E94-A, no. 10, pp. 1920–1927, Oct.
2011.
[19] L. Huang and T. Suzuki, “Weighted wavelet-based spectral-spatialtransforms for CFA-sampled raw image compression considering image
features,” in Proc. ICASSP, Singapore (Hybrid), May 2022, pp. 1850–
1854.
[20] MathWorks Help Center,“Implement Digital Camera
Processing
Pipeline,”
https://jp.mathworks.com/help/images/
end-to-end-implementation-of-digital-camera-processing-pipeline.
html, (Accessed on 06/06/2022).
[21] V. Bychkovsky, S. Paris, E. Chan, and F. Durand, “Learning photographic global tonal adjustment with a database of input/output image
pairs,” in Proc. CVPR, Colorado Springs, CO, Jun. 2011, pp. 97–104.

WWSST [19]
6.40
6.38
8.32
5.22
6.25
5.93
7.36
8.62
7.00
5.98
6.75
7.01
8.95
6.83
8.19
8.82
7.51
6.19
5.95
8.39
8.13
7.60
7.42
7.30
6.48
6.94
6.13
6.90
6.28
8.12
6.42
7.55
6.96

XSTT-I
6.40
6.40
8.32
5.19
6.28
5.91
7.35
8.65
6.94
6.01
6.75
6.88
9.01
6.77
8.17
8.83
7.48
6.15
5.96
8.37
8.08
7.57
7.41
7.27
6.53
6.94
6.19
7.00
6.30
8.09
6.57
7.55
6.99

XSTT-II
6.42
6.41
8.34
5.20
6.30
5.92
7.37
8.66
6.95
6.03
6.76
6.85
9.01
6.78
8.18
8.84
7.48
6.17
5.98
8.37
8.09
7.58
7.43
7.30
6.49
6.96
6.21
6.90
6.33
8.12
6.45
7.57
6.98

EXSTT-I
6.41
6.41
8.32
5.19
6.29
5.91
7.35
8.66
6.93
6.01
6.75
6.86
8.93
6.75
8.15
8.81
7.46
6.13
5.91
8.36
8.06
7.54
7.40
7.27
6.53
6.93
6.17
6.99
6.29
8.09
6.56
7.54
6.98

EXSTT-II
6.43
6.42
8.34
5.21
6.30
5.92
7.37
8.67
6.94
6.03
6.76
6.83
8.93
6.76
8.16
8.81
7.47
6.15
5.93
8.36
8.07
7.55
7.43
7.29
6.48
6.95
6.19
6.89
6.32
8.11
6.44
7.56
6.97

Taizo Suzuki (Senior Member, IEEE) received the
B.E., M.E., and Ph.D. degrees in electrical engineering from Keio University, Japan, in 2004, 2006, and2010, respectively. From 2006 to 2008, he was with
Toppan Printing Company, Ltd., Japan. From 2008
to 2011, he was a Research Associate of the Global
Center of Excellence (G-COE), Keio University,
Japan. From 2010 to 2011, he was a Research
Fellow of the Japan Society for the Promotion of
Science (JSPS) and a Visiting Scholar at the Video
Processing Group, University of California at San
Diego, La Jolla, CA, USA. From 2011 to 2012, he was an Assistant Professor
with Nihon University, Japan. In 2012, he joined the University of Tsukuba,
Japan as an Assistant Professor, where he has been an Associate Professor
since 2019. His current research interests include signal processing and its
application to media contents. From 2017 to 2021, he was an Associate Editorof the IEICE Transactions on Fundamentals of Electronics, Communications
and Computer Sciences.

Liping Huang (Student Member, IEEE) received
a B.Eng. degree in electrical and information engineering from Jimei University, China, in 2019. In
2019, she joined the University of Tsukuba, Japanas a Research Student, and since 2021, she has
been in the Master’s Program in Computer Science.
Her current research interest is image and video
processing.

11

TABLE VI
BD-PSNR S [ D B] IN LOSSY COMPRESSION .
Direct(0.75)(0.03)(2.33)(0.52)(−3.13)(−2.46)(−1.02)(−2.74)(0.50)(2.07)(−0.31)(1.97)(1.77)(0.46)(−0.74)(3.08)(0.73)(1.52)(0.85)(1.38)(0.86)(1.19)(0.67)(0.45)(−0.55)(−0.03)(0.95)(1.85)(2.72)(1.83)(2.70)(1.05)(1.16)

#0500
#1000
#1500
#2000
#2500
#3000
#3500
#4000
#4500
#5000
Avg.
#0092
#0482
#0548
#1137
#1145
#1463
#2239
#2912
#3394
#3419
Avg.
m01
m02
m03
m04
m05
m06
m07
m08
m09
m10
Avg.

MSST [10]
–
–
–
–
–
–
–
–
–
–
–
–
–
–
–
–
–
–
–
–
–
–
–
–
–
–
–
–
–
–
–
–
–

CAMRA [12]
0.37
0.31
0.91
0.25
0.21
0.53
0.71
0.24
1.11
0.39
0.50
1.34
1.60
0.93
0.80
0.53
−0.16
0.71
1.36
0.80
0.83
0.87
1.29
1.71
0.83
1.02
0.33
0.75
1.08
1.66
0.28
1.37
1.03

WSST [14]
1.38
1.31
1.49
1.17
1.23
1.34
2.01
1.48
1.67
1.56
1.46
1.97
2.31
1.82
1.39
1.09
0.60
1.34
2.24
1.87
1.69
1.63
2.13
2.33
2.03
1.94
1.24
1.97
1.97
2.55
1.89
2.61
2.07

STT [15]
1.35
1.07
1.43
1.18
0.67
1.24
1.76
1.31
1.92
1.09
1.30
2.93
2.17
1.67
1.35
1.00
0.82
1.28
2.08
1.73
1.66
1.67
1.99
2.12
1.70
1.70
0.77
1.52
1.61
2.25
1.45
2.29
1.74

WWSST [19]
1.19
1.11
0.65
1.09
1.08
0.00
2.02
1.26
1.58
1.45
1.14
−0.21
2.42
1.74
1.42
1.08
0.26
1.32
2.21
1.80
0.40
1.24
2.13
2.27
0.21
−0.87
0.74
0.11
1.85
2.58
0.52
2.64
1.22

XSTT-I
1.48
1.35
1.49
1.39
0.91
1.43
1.91
1.50
1.98
1.26
1.47
2.88
2.26
1.92
1.50
1.12
0.95
1.53
2.34
1.97
1.85
1.83
1.98
2.21
1.92
1.92
1.06
1.55
1.85
2.25
1.54
2.46
1.87

XSTT-II
1.24
1.02
1.40
1.27
0.58
1.25
1.72
1.19
1.89
1.19
1.27
3.21
2.34
1.82
1.42
1.06
1.04
1.55
2.27
1.88
1.71
1.83
2.04
2.17
1.90
1.79
0.88
1.71
1.75
2.46
1.81
2.45
1.90

EXSTT-I
1.22
0.99
1.08
1.09
0.57
0.62
1.39
0.72
1.83
1.11
1.06
2.26
2.02
1.86
1.43
1.06
0.74
1.50
2.31
−0.16
1.18
1.42
1.69
2.10
0.33
1.77
0.96
1.42
1.77
2.15
1.29
2.42
1.59

EXSTT-II
1.09
0.88
0.62
1.18
0.43
0.61
1.62
0.96
1.84
1.12
1.03
1.67
2.26
1.82
1.43
1.07
0.92
1.51
2.31
1.36
1.62
1.60
1.95
2.11
0.51
−1.03
0.46
0.33
1.68
2.40
0.65
2.46
1.15

54
52
PSNR [dB]

50
48
46
44
42
40
38

Difference in PSNR [dB]2.5
Direct for RGB
MSST [10]CAMRA [12]WSST [14]
STT [15]WWSST [19]
XSTT-I []XSTT-II []EXSTT-I []EXSTT-II []

2
1.5
1
0.5
0
2

2.5

3

3.5
4
Bitrate [bpp]

4.5

5

2

2.5

3

3.5
4
Bitrate [bpp]

4.5

5

2

2.5

3

3.5
4
Bitrate [bpp]4.5

5

Fig. 12. R-D curves of average bitrate and PSNR in lossy compression: (top) bitrate vs PSNR, (bottom) bitrate vs difference in PSNR compared with MSST
(i.e., assuming that the PSNRs of the MSST [10] are 0 dB), (left-to-right) general images, images with the many edges, and mobile phone images.

12Fig. 13. Part of images developed in lossy compression with 2 bpp: (top-to-bottom) original, XSTT-I, and XSTT-II and (left-to-right) #2000, #4500, #0482,#2239, m04, and m09.