1

Image Boundary Extension with Mean Value for
Cosine-Sine Modulated Lapped/Block Transforms
Ryoma Ishibashi, Taizo Suzuki, Senior Member, IEEE, Seisuke Kyochi, Member, IEEE, andHiroyuki Kudo, Member, IEEE

Abstract—We present a novel image boundary extension,mean value extension (MVE), for directional lapped transforms,
particularly cosine-sine modulated lapped transforms (CSMLTs).
Lapped transforms are usually used with an extension technique,such as periodic extension (PE) or symmetric extension (SE), fornonexpansive convolution at signal boundaries. When directional
textures (oblique lines and curves) appear at the 2D signal (image)boundaries, both PE and SE produce directional discontinuities,
which degrade the sparsity of the transformed coefficients, especially in the case of directional lapped transforms. MVE reduces
the discontinuities for directional textures better than PE or SE
do; it thus improves the efficiency of the sparse representation
based on directional lapped transforms. Moreover, to reduce
computational costs compared with those of directional lapped
transforms, we introduce new directional block transforms, called
cosine-sine modulated block transforms (CSMBTs). These newtransforms are derived from a minimum tile processing (tiling)of M -band CSMLTs with 2M filter lengths and nonexpansive
convolutions. The resulting directional block transforms, particularly in the case of the MVE, have richer directional selectivity
and have higher performance compared with a discrete Fouriertransform (DFT) as shown in experiments.
Index Terms—Cosine-sine modulation, directional block transforms, directional lapped transform, mean value extension, nonexpansive convolution.

I. I NTRODUCTION
ISCRETE cosine transforms (DCTs) [1] and discrete
wavelet transforms (DWTs) [2] are widely used forvarious types of image processing, e.g., image restoration(denoising, deblurring, and so on) and in the image/videocoding standard JPEG [3], the H.26x series [4], [5], and
JPEG 2000 [6]). However, since atoms (i.e., elements ofbasis/frames) of DWTs are lying only horizontal and verticaldirections (and components mixing ±45 degree directions);
they cannot directly express directional features such as
oblique lines and curves. Thus, they are often referred to as
having poor directional selectivity [7]; i.e., they cannot provide
efficient sparse representations for directional textures.
Directional transforms, which can express rich directional
textures sparsely, have been extensively studied. They areclassified into two classes. The first class, so-called “adaptive directional transforms” [8]–[12], typically does not haveD

R. Ishibashi is with NEC Corporation, Minato-ku, Tokyo, 108-8001 Japan(e-mail: r-ishibashi@bu.jp.nec.com).
T. Suzuki and H. kudo are with the Faculty of Engineering, Information and
Systems, University of Tsukuba, Tsukuba, Ibaraki, 305-8573 Japan (e-mail:{taizo, kudo}@cs.tsukuba.ac.jp).
S. Kyochi is with the Department of Information and Media Engineering,
The University of Kitakyushu, Kitakyushu, Fukuoka, 808-0135 Japan (e-mail:s-kyochi@kitakyu-u.ac.jp).

directional atoms except for vertical and horizontal directions(as in DCT and DWT), but they are applied to images alongpredetermined oblique directions (the direction can be changed
at every pixel). However, the applications of adaptive directional transforms are relatively limited because, for example inimage recovery, it is difficult to determine suitable directions
from a degraded input image contaminated by noise, blur,
pixel missing, and so on. Another one is a class of transformsof which atoms are lying along various directions. Since
the applications of the directional transforms with directional
atoms cover general applications including signal recovery, we
focus on them and will call them “directional transforms.”
They are further classified into directional lapped transforms(such as curvelet [13], ridgelet [14], and contourlet [15], andthe multidimensional nonseparable lapped transforms [16]–[18]) and directional block transforms (such as discrete Fouriertransform (DFT) [19]). Directional lapped transforms are
directional transforms with the atoms of which supports arelapped each other, while directional block transforms are
directional transforms with the atoms of which supports are
not lapped.
Among them, dual-tree complex wavelet transforms(DTCWTs) [7] have been paid much attention to because of
their low computational complexity and rich directional selectivity. Their directional functions in the frame are generated
by using two parallel maximally decimated filter banks (FBs)
satisfying the half-sample delay condition [20]. One problem
with DTCWTs is that, since the half-sample delay condition
is difficult to approximate with finite impulse response (FIR)
filtering, the resulting filter performance characteristics, suchas stopband attenuation and coding gain, are often degraded,especially with short filter lengths. To solve this problem,
cosine-sine modulated lapped transforms (CSMLTs)1 havebeen widely studied [21]–[24] as an alternative class of
DTCWTs.2 CSMLTs can be easily designed by modulatinga prototype lowpass filter instead of approximating the halfsample delay condition, and they work better than DTCWTs
do in image processing, e.g., image denoising [23], [24].
The CSMLTs are lapped transforms which have longer
1 Although they are often called cosine-sine modulated FBs (CSMFBs), we
will call them CSMLTs to clearly distinguish them from block transforms.
2 Modulated complex lapped transform (MCLT) in [25] is essentially thesame transform as the CSMLTs because they are also designed by the modulation of a prototype filter (the definition in the cosine and sine modulation
terms is slightly different). However, the MCLT has been introduced as a 1D
complex-valued transform for effective audio processing in [25]. On the other
hand, the CSMLTs has been firstly introduced as a directional selective 2D
real-valued transform for effective image processing in [21].

2filter lengths L than the number of bands (or band count)
M (M, L ∈ N, L > M ) and require extra signals for
perfect reconstruction (PR) at boundaries. Nonexpansive convolution, e.g., periodic extension (PE) and symmetric extension (SE) [26], is one of the simplest approaches to solving the boundary problem. PE often causes discontinuities
at boundaries, whereas SE typically extends the boundaries
more smoothly. There are also other smooth nonexpansive
convolutions for various FBs [27]–[29]. However, when theconventional nonexpansive convolution methods are applied
to images involving directional textures at boundaries, “directional” lapped transforms such as CSMLTs cannot provide
efficient sparse representations at image boundaries. Thus, inimage restoration for example, this often results in annoying
artifacts near the boundaries.
Here, we present a nonexpansive convolution, called meanvalue extension (MVE), that can simply reduce discontinuities
at boundaries by extending with the mean value of the pixels
located near the boundaries. Unlike PE and SE, this extension
requires a lattice structure implementation of FBs for its execution. Since, as will be illustrated in this paper, the CSMLTs
can be designed with a lattice structure implementation [30],
MVE can be incorporated with them.
We further introduce new directional block transforms,
called cosine-sine modulated block transforms (CSMBTs),
derived from a minimum tile processing (tiling) of CSMLTs
with 2M filter lengths and nonexpansive convolutions. Blocktransforms with a filter length L equal to the number ofbands M (L = M ) generally have lower computationalcosts than lapped transforms do, and their computations forevery block in an input image can be fully parallelized. The
DCT and discrete sine transform (DST) [1] are efficient blocktransforms, and they are used in many applications, such as
JPEG and the H.26x series. However, they cannot analyze
various directions because, like DWTs, their basis suffers frompoor directional selectivity. On the other hand, the DFT is
the most-used “directional” block transform. One problem
with the DFT is that some atoms in its basis have the same
directions. Thus, the number of directions is limited. Since
the new CSMBTs, especially in the case of MVE, have richer
directional selectivity, they are expected to work better thanthe DFT in various image processing applications.
A preliminary version of this paper was presented in [31],where we discussed only the MVE for M -band CSMLTs witha filter length of 2M . This paper describes the general case of afilter length L longer than the number of bands M . Moreover,it discusses how the MVE is used in the design of CSMBTs.
The remainder of the paper is organized as follows. Section II reviews the traditional directional block transforms
and CSMLTs. Section III presents MVE and derives the
CSMBTs. Section IV compares the resulting transforms withthe conventional methods. Section V concludes this paper.
Notation: Boldface lower- and upper-case letters represent
vectors and matrices, respectively. The number of bands M istechnically constrained to an even number. The set of M × N
real-valued matrices is denoted as RM ×N . For A ∈ RM ×M ,[A]× ∈ R 2 × 2 (× = α, β, γ, orδ) is the submatrix,[][A]α [A]βA=.
[A]γ [A]δ
M

M

(1)

A⊤ is the transpose of the matrix A. diag(a0 , · · · , aM −1 )and diag(A0 , · · · , AM −1 ) are M × M diagonal and block
diagonal matrices whose diagonal elements are a0 , · · · , aM −1
and A0 , · · · , AM −1 , respectively. I, J, and O indicate theidentity matrix, the reversal identity matrix, and the zeromatrix, respectively. A diagonal matrix D is defined as√D =diag(1, −1, · · · , 1, −1). j is the unit complex number −1.
ec is the conjugate
of c ∈ C. H(z) ∑and H(z)are defined as∑−neH(z) = n h(n)zand H(z)= n h(n)z n , respectively.
H(ω) = H(ejω ). An M × L transform is a transform withnumber of bands M and filter length L.
II. R EVIEW AND D EFINITIONS
A. Directional Block Transforms
The DFT [19] is expressed for an M -sample finite discrete
signal x(n) as
M −1

2πmn
1 ∑
X(m) = √x(n)e−j M
M n=0

for 0 ≤ m ≤ M − 1.
(2)

b = F x,This transform can be equivalently expressed as x
b are M ×1 input and output signals, respectively,
where x and x
and the (m, n)th-element of the M -band DFT matrix F isdefined as
2πmn
1[F ]m,n = √ e−j M .
(3)
M
The DCT and DST [1] come in various types. The (m, n)thelements of the M -band type-II DCT (DCT-II) matrix C II ,which is used in JPEG and the H.26x series, type-II DST(DST-II) matrix S II , type-IV DCT (DCT-IV) matrix C IV , andtype-IV DST (DST-IV) matrix S IV are respectively defined as√2πm(n + 1/2)
[C II ]m,n =sm cos
(4)
M
M
√
2 ′
π(m + 1)(n + 1/2)
[S II ]m,n =sm sin(5)MM√2π(m + 1/2)(n + 1/2)
cos
(6)
[C IV ]m,n =MM√
2π(m + 1/2)(n + 1/2)
[S IV ]m,n =
sin
,
(7)
M
M
where
{
√1
(m = 0)2sm =1(m ̸= 0){1√(m = M − 1)
2
s′m =
.
1
(m ̸= M − 1)
The separable 2D DCT and DST do not exhibit directional(m1 ,m2 )orientation. For example, each 2D DCT-II atom Cn1 ,n
2
forms
2
1 ,m2 )cos ϑm1 ,n1 cos ϑm2 ,n2 ,(8)Cn(m= sm1 sm2
1 ,n2
M

3

(a)

(b)

Fig. 1: Atoms in basis (M = 4): (a) DCT-II, (b) DFT (realpart).

Fig. 3: Time-series processing of lattice structure of M -band
CMLT (arrows indicate M/2 signals).

half-sample delay condition. Instead, they are constructed fromtwo M -band parallel FBs, which are designed by performing
cosine and sine modulations on a prototype filter p(n), i.e., byapplying cosine and sine modulated lapped transforms (CMLT
and SMLT).
1) CMLT and SMLT: The CMLT and SMLT are shown in′
Fig. 2 [32]. The filter coefficients of Hm (z), Fm (z), Hm(z),′
and Fm (z) (0 ≤ m ≤ M − 1) are expressed as follows [21]:Fig. 2: M -band CSMLTs.
where m1 and n1 denote vertical subband/spatial indices, m2
and n2 denote horizontal ones (m1 , m2 , n1 , n2 ∈ {0, · · · , M −1}), andπm(n + 1/2).
M
Fig. 1(a) shows examples of DCT atoms.3 Clearly, they are notdirectionally oriented and cannot express various directional(m ,m )
components. On the other hand, each 2D DFT atom Fn1 ,n1 2 2
forms
1 j (φm1 ,n1 +φm2 ,n2 )
1 ,m2 )
Fn(m
=
e
,
(9)
1 ,n2
M
where
2πmn
φm,n =.
M
As shown in Fig. 1(b), the basis contains directionally oriented
atoms. One problem with the 2D DFT is that its basis
contains duplicate atoms and it cannot provide rich directional
selectivity, as shown in Fig. 1(b). This follows from the factthat
ϑm,n =ej (φm1 ,n1 +φm2 ,n2 ) = ej (φM −m1 ,n1 +φM −m2 ,n2 ) .

(10)

Hence, when used in image analysis and processing, DFTs
often degrade the efficiency of image analysis and processing.

B. Cosine-Sine Modulated Lapped Transforms (CSMLTs)
CSMLTs are an alternative class of DTCWTs with rich directional selectivity. Unlike DTCWTs, they do not require the
3 In Fig. 1, each atom is enlarged for visualization.

where

hm (n) = 2p(n) cos (ϕm,n + ψm )(11)

fm (n) = 2p(n) cos (ϕm,n − ψm )h′m (n) = 2p(n) sin (ϕm,n + ψm )′(n) = 2p(n) sin (ϕm,n − ψm ) ,
fm

(12)
(13)
(14)

(

)
(
)
1 π
L−1
ϕm,n = m +
n−
2 M
2
mπ
ψm = (−1)
4and L is the length of the prototype filter p(n). The conditionon the prototype filter p(n) for PR is [32]e ℓ (z)Gℓ (z) + G
e M +ℓ (z)GM +ℓ (z) = 1
G
( 2M
)
M
0≤ℓ≤
−1 ,
2
(15)

where Gℓ (z) (0 ≤ ℓ ≤ 2M −1) are the polyphase componentsof the prototype filter P (z), i.e.,
P (z) =2M
−1∑

Gℓ (z 2M )z −ℓ ,

ℓ=0

where
Gℓ (z) =∑p(2M n + ℓ)z −n .

n

The CMLT and SMLT with the prototype filter p(n) can be
represented as the following lattice structure of the polyphasematrices by using the symmetry of cosine/sine modulation
terms cos(ϕm,n ±ψm ) and sin(ϕm,n ±ψm ) in (11)-(14) and the
PR condition in (15). Let EC (z) and ES (z) be the polyphasematrices of an M × 2KM (K ∈ N) CMLT and SMLT,
respectively. The relationship between two polyphase matrices

4

fC (z), ℓ ∈ Z. If the CMLT satisfies the PR
is ES (z) = z −ℓ DE
condition, the SMLT also satisfies it. The polyphase matrix
EC (z) based on DCT-IV C IV can be expressed as a lattice
structure (see Fig. 3) [30]:
(K−1
)
∏ ( )
2
EC (z) = C IV WΛ(z)Θ0Λ z Θk J,
(16)
k=1

where

[

]

[

]

[ −1
]
−Ci Si J
O I
z I O
Θi =
, W=
, Λ(z) =JSi JCi J
I O
O
I
(
)
Ci = diag cos θi,0 , . . . , cos θi,M/2−1()
Si = diag sin θi,0 , . . . , sin θi,M/2−1(a)and θi is a free parameter for 0 ≤ i ≤ K − 1. Let
ΞK−1 ≜ ΘK−1 J(17)

for later discussion. Another polyphase matrix ES (z) basedon DST-IV S IV is
(K−1
)
∏ ( )
′
′
2
ES (z) = S IV WΛ(z)Θ0Λ z Θk J,
(18)
k=1

where
Θ′i =

[

Ci
JSi

] [
]
[
]
Si J
I O
−I O
=
Θi
.
−JCi J
O −I
O I

2) CSMLTs as 2D Directional Lapped Transform: The
directional selectivity arising from CSMLTs can be explained
as follows. Let Um (ω) and Vm (ω) be a complex combination′of spectra between Hm (ω) and Hm
(ω),
′
(ω)
Um (ω) = Hm (ω) + jHm

=e

jηk

L−1
∑

p(n)e

π
−j (ω−(m+ 12 ) M
)n

(
(
) )
1 π
ω− m+
,
2 M

1
′
(Hm (ω) − jHm
(ω))
j
(
(
) )
1 π
−jηk
=e
P ω+ m+
,
2 M

(19)

Vm (ω) =where

Fig. 4: Frequency responses of 4 × 8 CSMLTs′′([0, 2π]): (a) Hm (ω) + jHm(ω) and Hm (ω) − jHm(ω),(b)
Um1 (ω1 )Um2 (ω2 )+
Vm1 (ω1 )Vm2 (ω2 )andUm1 (ω1 )Vm2 (ω2 )+
Vm1 (ω1 )Um2 (ω2 ),wherem, m1 , m2 ∈ {0, 1, 2, 3}.
concentration [33], are shown in Fig. 4(a). From (19) and (20),
we can derive the following 2D frequency spectra:
Um1 (ω1 )Um2 (ω2 ) + Vm1 (ω1 )Vm2 (ω2 )′′= Hm1 (ω1 )Hm2 (ω2 ) − Hm(ω1 )Hm(ω2 )
12
Um (ω1 )Vm (ω2 ) + Vm (ω1 )Um (ω2 )′′= Hm1 (ω1 )Hm2 (ω2 ) + Hm
(ω1 )Hm(ω2 ).
12

n=0= ejηk P

(b)

(20)

(
)
1 π L−1πηk = − m ++ (−1)m .
2 M 2
4

Since Um (ω) and Vm (ω) are the modulations of a prototype
lowpass filter P (ω), each frequency response is one-sided
in the positive or negative frequency domain. The frequency
responses of Um (ω) and Vm (ω) in the case of
(
))
(
1
π
n+
(21)
p(n) = − sin2M2
for 0 ≤ n ≤ 2M − 1, which satisfies the PR condition in(15) and generates a lapped transform with maximum DC

(22)
(23)

This indicates that directional spectra of the (1,3)-quadrantand (2,4)-quadrant can be generated by performing additions
and subtractions between the transform coefficients obtainedby the separable 2D CMLT and SMLT, as shown in Fig.
4(b). The construction of the 2D directional CSMLTs andits corresponding 2D directional basis functions after theoperation
are shown in Fig. 5. Although the scaling coefficient√
1/ 2 has been omitted in the equations, it is actually requiredfor 2D directional CSMLTs to be a tight frame, i.e., energy
preserving transform.
III. N ONEXPANSIVE C ONVOLUTION FOR CSMLT S ANDCSMBT S
When a lapped transform is applied to an image, a nonexpansive convolution should be used at the boundaries not
to increase the number of samples. In particular, SE is often
used to avoid discontinuities at the boundaries. However, when
SE is used on directional lapped transforms such as CSMLTs,
directional discontinuities appear and the boundaries are distorted by annoying boundary artifacts in image restoration.
Here, we derive an efficient nonexpansive convolution for
CSMLTs. Tanaka et al. devised a SE for nonlinear-phase5in K ≥ 2. At the bottom-right of Fig. 6, we reconstruct x0 ,x1 , and x2 . x1 and x2 are reconstructed from b0 and b5 as
[ ]
[ ]
x1
⊤ b0
= Ξ1
,
(26)
x2
b5
where Ξ1 is in (17). Although b0 is not directly available atthe inverse transform side, we can compensate it as follows.
At the forward transform side, b1 is calculated by
[ ]
[
] x2
b1 = [Ξ1 ]γ E [Ξ1 ]δ E
.
(27)
x1
Fig. 5: Construction of M ×2KM CSMLTs and corresponding
2D directional basis functions of 4 × 8 CSMLTs.

Accordingly, we substitute (26) in (27), from which (27) is
calculated as[ ][] ⊤ b0b1 = [Ξ1 ]δ E [Ξ1 ]γ E Ξ1
.
(28)
b5

paraunitary FBs (NLPPUFBs); they used a boundary filtering
technique [27]. We will extend this idea to our nonexpansive
convolution. Moreover, we propose new directional blocktransforms, called CSMBTs, derived from M × 2M CSMLTs
with the nonexpansive convolutions.

On the other hand, a0 at the forward transform side iscalculated as
[ ]
[
] b0
a0 = [Θ0 ]γ [Θ0 ]δ
.
(29)
b1

A. Nonexpansive Convolution for CSMLTs

Therefore, at the inverse transform side, b0 can be calculated
by solving the simultaneous matrix equation with (28) and(29). We obtain

Here, we develop a nonexpansive convolution by boundary
filtering for CMLT in accordance with [27]. The derivation for
SMLT is omitted because it can be verified in the same way.
Fig. 6 shows the upper boundary processing in the case of
K = 1 and 2. The boundary signals are generated by using
a boundary extension matrix E M , e.g., E M = J M in the222
case of SE. Given an arbitrary K, the upper and lower input
signals require (2K − 1)M extra signals because of the filter
length L = 2KM for CSMLTs. We will show derivations fortwo cases, i.e., K = 1 and 2; we can easily handle the casesof K ≥ 3 in a similar way by solving simultaneous matrix
equations.
1) Case of K = 1: Signal reconstruction in the case of
K = 1 is achieved with M extra signals in total. We consider
the case of the boundary processing when reconstructing x0
from a0 at the top-right of Fig. 6. At the top-left of Fig. 6,
the signal a0 is calculated as
[ ]
[
] x0
a0 = [Ξ0 ]γ E [Ξ0 ]δ
,
(24)
x0
M
where Ξ0 is in (17) and E is an arbitrary M
2 × 2 boundary
extension matrix. We solve for x0 as follows:

x0 = U −1
0 a0 ,(25)

where
U 0 = [Ξ0 ]γ E + [Ξ0 ]δ .
In design of the CSMLTs, the coefficients in Ξ0 are heuristically chosen such that det(U 0 ) ̸= 0. The lower boundaries
can be calculated in the same way as the upper boundaries.
2) Case of K = 2: Signal reconstruction in the case of
K = 2 is achieved with 3M extra signals in total. Since thedelay elements of CSMLTs in K ≥ 2 are z −2 unlike z −1 in
NLPPUFBs, we have to recalculate the boundary processing

b0 = V −1
0 (a0 − W 0 b5 ) ,

(30)

where

(
)
⊤
⊤
V 0 = [Θ0 ]δ [Ξ1 ]δ E [Ξ1 ]α + [Ξ1 ]γ E [Ξ1 ]β + [Θ0 ]γ ,()⊤⊤
W 0 = [Θ0 ]δ [Ξ1 ]δ E [Ξ1 ]γ + [Ξ1 ]γ E [Ξ1 ]δ .
As for x0 , we can reconstruct from the same discussion
explained in the case of K = 1:x0 = U −1
1 b3 ,(31)

where
U 1 = [Ξ1 ]γ E + [Ξ1 ]δ .
Similar to the case of K = 1, the coefficients are heuristicallychosen in design such that det(U 1 ) ̸= 0 and det(V 0 ) ̸= 0.
The lower boundaries can be calculated in the same way as
the upper boundaries.
B. Mean Value Extension (MVE)
Before introducing MVE, we should confirm that the conventional extension methods are inefficient for directional
textures. Figs. 7(b) and (c) show the results of PE and SE for
the input image in Fig. 7(a). Since PE extends the directional
textures at the boundaries along different directions, severe
discontinuities appear. Although SE can extend them smoothly
in terms of 1D signals, reflected (inverse) directional texturesappear at the boundaries.
To reduce the directional discontinuities, we propose amore appropriate boundary extension matrix E for CSMLTs.
Because of the separable implementation, not all of the directional textures can be extended smoothly. Because of that,

6

Fig. 6: Upper boundary processing of M -band CMLTs (each arrow indicates M/2 input signals): (top) K = 1; (bottom)
K = 2.

(a)

(b)

(c)

(d)

Fig. 7: Boundary extensions: (a) original image (zero extension), (b) PE, (c) SE, and (d) MVE.

block transforms, called CSMBTs. They can be derived from aminimum tiling of M × 2M CSMLTs, i.e., the case of K = 1
in Section III-A1, and the nonexpansive convolutions. Notethat the CSMBTs cannot be obtained when K ≥ 2 because
the minimum tile size is always bigger than M .
The CMLT with the nonexpansive convolution when K = 1
for input signals longer than M is expressed as
diag (C IV , C IV , · · · , C IV )× diag (JC0 E + JS0 J, Ξ0 , Ξ0 , · · · , Ξ0 , −C0 JE + S0 ) .
(33)
When the size of input signals is M , i.e., the minimum tiling,(33) takes the following form (see the bottom of Fig. 8):[]
JC0 E + JS0 J
O
BC = C IV.
(34)O−C0 JE + S0

Fig. 8: Minimum tiling of CMLT: (top) without a nonexpansive
convolution; (bottom) with a nonexpansive convolution.
we use the mean values of signals at the boundaries; i.e., theMVE matrix E is given byE=

2
1M M ,
M 2×2

(32)

M
where 1 M × M is an M
2 × 2 matrix with 1s in every element.
22
As shown in Fig. 7(d), MVE can extend the lines at eachboundary more smoothly so that severe directional discontinuities do not appear.

C. Cosine-Sine Modulated Block Transforms (CSMBTs)
Block transforms are used more than lapped transforms insmall-scale embedded systems to keep down computationalcosts and gain benefit from parallel computation, e.g., they
are used in image and video coding standards such as JPEG
and the H.26x series. Here, we present new M ×M directional

It is clear that BC is an M ×M block transform; we thus term
it the cosine modulated block transform (CMBT). An M × M
sine modulated block transform (SMBT) based on M × 2M
SMLT is similarly obtained as[]−JC0 E + JS0 J
O
BS = S IV.
(35)O
C0 JE + S0
Finally, the 2D directional CSMBTs are constructed by using
the CMBT and SMBT instead of the CMLT and SMLT in
Fig. 5. The Appendix shows a concrete numerical example of
the relationship between 4 × 8 CSMLTs and 4 × 4 CSMBTs
with MVE.
Remark: When the boundary extension matrix E is acentrosymmetric matrix that satisfies EJ = JE, such as SE
and MVE matrices, SMBT BS has a simple relationship with
CMBT BC :
[
]
[
]
J∆1 J
O
∆0 O
BS = DC IV J
= DC IV
J
O
J∆0 J
O ∆1
= DBC J,
(36)

7

(a)

(b)

(c)

(d)

(a)

Fig. 10: 2D directional basis functions of 8 × 8 block transforms: (a) DFT, (b) CSMBTs-PE, (c) CSMBTs-SE, and (d)CSMBTs-MVE.
(b)(c)
Fig. 9: Frequency responses of 8 × 8 CSMBTs ([0, 2π]): (a)′PE, (b) SE, and (c) MVE: (left) Hm (ω) + jHm(ω); (right)′
Hm (ω) − jHm (ω), where m ∈ {0, 1, · · · , 7}.
where∆0 = JC0 E + JS0 J∆1 = −C0 JE + S0 .
This relationship is the same as that holds between DCT-IV
and DST-IV, S IV = DC IV J.
IV. E XPERIMENTAL R ESULTS

in Optimization ToolBox of MATLAB. Moreover, wederived 8 × 8 CSMBTs from 8 × 16 CSMLTs with nonexpansive convolutions, which are PE, SE, and MVE. In thissection, the CSMLTs and CSMBTs with PE, SE, and MVE arelabeled CSMLTs-PE, CSMLTs-SE, CSMLTs-MVE, CSMBTsPE, CSMBTs-SE, and CSMBTs-MVE.
The complex combinations of the frequency responses and
2D directional basis functions of the 8 × 8 directional blocktransforms are shown in Figs. 9 and 10, respectively. Although
the spectra based on CSMBTs-PE in Fig. 9(a) are onesided in the positive or negative domain, as are the CSMLTs
in Fig. 4(a), some spectra are completely duplicated. Thus,
the resulting atoms from the CSMBTs-PE contain duplicate
directional ones, as shown in Fig. 10(b). On the other hand,whereas none of the spectra based on CSMBTs-SE (Fig. 9(b))
are duplicated, they do not sufficiently attenuate in eachstopband. Therefore, as shown in Fig. 10(c), each atom cannot
distinguish the corresponding direction accurately. By contrast,
the spectra from the CSMBTs-MVE are not duplicated andsufficiently attenuate in each stopband; the resulting atoms
have richer directional selectivity than the others.
B. Non-linear Approximation (NLA)

where BIT and MSE are the image’s bit-depth and meansquared error, respectively. The image set included 512 × 512
8-bit standard grayscale images in [34] such as Barbara,
Goldhill, and Lena.

We applied the CSMLTs, CSMBTs, and DFT to the nonlinear approximation (NLA) [36], which is often used as acriterion for efficiency of sparse representation. Fig. 11 shows
the rate-distortion (R-D) curves of NLA. The CSMLTs-MVE
and CSMBTs-MVE achieved the best performance in everycase. The top images of Figs. 12 and 13 show particularareas of Barbara and Lena in NLA (coefficient = 4000).
The PE/SE-based directional transforms generated annoying
artifacts, such as boundary artifacts, directional block artifacts,
and black/white block artifacts, whereas CSMLTs-MVE andCSMBTs-MVE did not generate any.

A. Filter Design

C. Image Denoising

We designed 8 × 16 and 8 × 32 CSMLTs by using thecost function, which is a weighted linear combination of
the coding gain and the DC leakage [35], and fminunc.m

We also compared the CSMLTs, CSMBTs, and DFT by
applying them to image denoising. Noisy images were madeby adding Gaussian random noise with a standard deviation

This section evaluates the CSMLTs and CSMBTs with PE,
SE, and MVE. The resulting transforms were compared interms of the peak signal-to-noise ratio (PSNR) [dB]:(()2 )2BIT − 1
PSNR[dB] = 10 log10
,
MSE

8

(a)

(b)

(c)

(d)

(e)

(f)

Fig. 11: R-D curves of NLA: (a-c) CSMLTs, (d-f) CSMBTs, (a,d) Barbara, (b,e) Goldhill, and (c,f) Lena.

Fig. 12: Particular areas (left sides are boundaries) of Barbara
in NLA (coefficients = 4000) and image denoising (σ = 30)
by CSMLTs: (top) NLA, (bottom) image denoising, (leftto-right) original/noisy images, 8 × 16 CSMLTs-PE, 8 × 16
CSMLTs-SE, 8×16 CSMLTs-MVE, 8×32 CSMLTs-PE, 8×32
CSMLTs-SE, and 8 × 32 CSMLTs-MVE.

Fig. 13: Particular areas of Lena in NLA (coefficients = 4000)
and image denoising (σ = 10) by 8 × 8 directional blocktransforms: (top) NLA, (bottom) image denoising, (left-toright) DFT, CSMBTs-PE, CSMBTs-SE, and CSMBTs-MVE.

V. C ONCLUSION

σ to the original images. The √images were denoised byapplying a universal threshold σ 2 log N [37], where N is
the number of pixels, after the transforms had been performed.
Fig. 14 shows the R-D curves of image denoising. CSMLTsMVE and CSMBTs-MVE performed the best in almost allcases. Moreover, the bottom images of Figs. 12 and 13 show
particular areas of Barbara and Lena in image denoising(σ = 30 and 10). Similar to the case of NLA, the PE/SEbased directional transforms generated annoying artifacts, but
CSMLTs-MVE and CSMBTs-MVE did not generate any.

We proposed a new nonexpansive convolution, MVE, for
CSMLTs. The conventional boundary extension methods, PE
and SE, often produce directional discontinuities at boundaries, which degrade the efficiency of a sparse representation
created by using a directional lapped transform. MVE reduces
directional discontinuities by using the mean value of the pixels near the boundaries. We also devised new directional blocktransforms, called CSMBTs, derived from the minimum tilingof the CSMLTs with a 2M filter length and the nonexpansive
convolutions. The resulting CSMBTs-MVE not only have richdirectional selectivity at a low cost but also better mitigate
annoying artifacts in NLA and image denoising compared with
the DFT.

9

(a)

(b)

(c)

(d)

(e)

(f)

Fig. 14: R-D curves of image denoising: (a-c) CSMLTs, (d-f) CSMBTs, (a,d) Barbara, (b,e) Goldhill, and (c,f) Lena.

A PPENDIXA C ONCRETE N UMERICAL E XAMPLE OF R ELATIONSHIPBETWEEN CSMLT S AND CSMBT S WITH MVE

This Appendix shows a concrete numerical example ofrelationship between 4 × 8 CSMLTs and 4 × 4 CSMBTs
with MVE. The resulting transforms are shown in Table I.
We consider a pseudo-random 4 × 4 input block x as

3
5
x=
5
9

1
9
3
7

4
2
5
9


1
6

8
3

(37)

and the 8 × 8 MVE-based extended block x′ as

4.5 4.5 4 5 3 3.5 3.25 3.25
4.5 4.5 4 5 3 3.5 3.25 3.25

2
2 3 1 4 1
2.5
2.5 


7
7 5 9 2 6
4
4 
′

.
x =
4 5 3 5 8
6.5
6.5 
4

8
8 9 7 9 3
6
6 


6
6 7 5 7 5.5 6.25 6.25
6
6 7 5 7 5.5 6.25 6.25

When 4 × 8 CSMLTs and 4 × 4 CSMBTs are applied to x′
and x, we obtain the 2D transformed coefficients as[] []⊤EC0 EC1 x′ EC0 EC1 = BC xB⊤C
17.27620.8656 −0.5848 −1.4899
−3.5692 −0.0174 −0.6653 1.2170  ≜ yC (39)=
−1.5671 −0.7004 1.3007
4.5910 −5.3497 −1.2981 2.7250 −0.0447[] []⊤ES0 ES1 x′ ES0 ES1 = BS xB⊤
S
23.2061
0.3670 −0.1711 1.4449
−2.6012 −1.0096 2.2133 −3.1325=
−1.9793 2.9281 −2.0320 0.8948  ≜ yS ,−1.6648 −2.3473 −1.3771 4.0055(40)
respectively. They mean that the CSMBTs with MVE are exactly the same as a minimum tiling of the CSMLTs with MVE.
Note that the resulting output√ directional√ signals in the 2D
transforms are (yC + yS )/ 2 and (yC − yS )/ 2.
ACKNOWLEDGMENT(38)

The authors would like to thank the anonymous reviewers
for providing many constructive suggestions that significantly
improve the presentation of this paper. This work was supported by Grant-in-Aids for Young Scientists (B), Grant Number 16K18100 and 17K14683, from the Japan Society for the
Promotion of Science (JSPS).
R EFERENCES[1] K. R. Rao and P. Yip, Discrete Cosine Transform Algorithms, Academic
Press, 1990.

10

TABLE I: Concrete filter coefficients of resulting transforms (M = 4).
Resulting Transforms
4 × 8 CMLT EC (z)

0.1147
−0.0269
−0.1353−0.0766|


4 × 8 SMLT ES (z)0.0766
 0.1353
 0.0269−0.1147|

0.38530.32660.21830.0766{z

0.5766
0.48890.3266
0.1147≜EC00.07660.21830.3266
0.3853−0.1147−0.3266−0.4889−0.5766{z≜ES0

4 × 4 CMBT BC

4 × 4 SMBT BS

Filter Coefficients 
0.5766
0.38530.1147−0.0766−0.1353 −0.6802 −0.3266
0.2183+−0.6802  0.1353
0.4889−0.3266−0.3853
0.5766−0.5766
0.3853} |{z −0.3853−0.5766−0.6802 −0.1353+−0.13530.68020.5766−0.3853} |
0.8266
0.6387
0.36810.1147

0.0381
0.1499
0.31210.4413[2] G. Strang and T. Nguyen, Wavelets and Filter Banks, WellesleyCambridge Press, 1996.
[3] G. K. Wallace, “The JPEG still picture compression standard,” IEEE
Trans. Consum. Electr., vol. 38, no. 1, pp. xviii–xxxiv, Feb. 1992.
[4] T. Wiegand, G. J. Sullivan, G. Bjøntegaard, and A. Luthra, “Overview
of the H.264/AVC video coding standard,” IEEE Trans. Circuits Syst.
Video Technol., vol. 13, no. 7, pp. 560–576, July 2003.
[5] G. J. Sullivan, J.-R. Ohm, W.-J. Han, and T. Wiegand, “Overview of thehigh efficiency video coding (HEVC) standard,” IEEE Trans. Circuits
Syst. Video Technol., vol. 22, no. 12, pp. 1649–1668, Dec. 2012.
[6] A. Skodras, C. Christopoulis, and T. Ebrahimi, “The JPEG2000 still
image compression standard,” IEEE Signal Process. Mag., vol. 18, no.
5, pp. 36–58, Sep. 2001.
[7] I. W. Selesnick, R. G. Baraniuk, and N. C. Kingsbury, “The dual-treecomplex wavelet transform,” IEEE Signal Process. Mag., vol. 22, no.
6, pp. 123–151, June 2005.
[8] W. Ding, F. Wu, X. Wu, and S. Li, “Adaptive directional lifting-based
wavelet transform for image coding,” IEEE Trans. Image Process., vol.
16, no. 2, pp. 416–427, Feb. 2007.
[9] H. Xu, J. Xu, and F. Wu, “Lifting-based directional DCT-like transform
for image coding,” IEEE Trans. Circuits Syst. Video Technol., vol. 17,
no. 10, pp. 1325–1335, Oct. 2007.
[10] J. Xu, F. Wu, J. Liang, and W. Zhang, “Directional lapped transforms
for image coding,” IEEE Trans. Image Process., vol. 19, no. 1, pp.
85–97, Jan. 2010.
[11] T. Yoshida, T. Suzuki, S. Kyochi, and M. Ikehara, “Two dimensional
non-separable adaptive directional lifting structure of discrete wavelet
transform,” IEICE Trans. Fundamentals., vol. E94-A, no. 10, pp. 1920–
1927, Oct. 2011.
[12] F. A. B. Hamzah, T. Yoshida, M. Iwahashi, and H. Kiya, “Adaptive
directional lifting structure of three dimensional non-separable discrete
wavelet transform for high resolution volumetric data compression,”
IEICE Trans. Fundamentals., vol. E99-A, no. 5, pp. 892–899, May 2016.
[13] J. L. Starck, E. J. Cand‘es, and D. L. Donoho, “The curvelet transform
for image denoising,” IEEE Trans. Image Process., vol. 11, no. 6, pp.
670–684, June 2002.
[14] M. N. Do and M. Vetterli, “The finite ridgelet transform for image
representation,” IEEE Trans. Image Process., vol. 12, no. 1, pp. 16–28,
Jan. 2003.
[15] M. N. Do and M. Vetterli, “The contourlet transform: An efficient
directional multiresolution image representation,” IEEE Trans. Image
Process., vol. 14, no. 12, pp. 2091–2106, Dec. 2005.
[16] S. Muramatsu, T. Kobayashi, M. Hiki, and H. Kikuchi, “Boundary
operation of 2-D nonseparable linear-phase paraunitary filter banks,”
IEEE Trans. Image Process., vol. 21, no. 4, pp. 2314–2318, Apr. 2012.
[17] S. Muramatsu, D. Han, T. Kobayashi, and H. Kikuchi, “Directionallapped orthogonal transform: Theory and design,” IEEE Trans. Image
Process., vol. 21, no. 5, pp. 2434–2448, May 2012.
[18] S. Muramatsu, K. Furuya, and N. Yuki, “Multidimensional nonseparable
oversampled lapped transforms: Theory and design,” IEEE Trans. Signal
Process., vol. 65, no. 5, Mar. 2017.

0.82660.0146−0.6387−0.3853
0.30870.5034−0.0415−0.71190.3087−0.5034−0.04150.71190.8266−0.0146−0.6387
0.3853≜EC1−0.5766
0.4889−0.3266
0.1147{z−0.3853
0.3266−0.2183
0.0766−0.0766
0.1353  −1z−0.0269
−0.1147}
−0.1147−0.0269 −1z0.1353 −0.0766}≜ES1

0.0381−0.1499
0.3121 −0.4413
0.8266−0.6387
0.3681 −0.1147[19] N. Ahmed and K. R. Rao, Orthogonal Transforms for Digital Signal
Processing, Berlin, Germany: Springer-Verlag, 1975.
[20] I. W. Selesnick, “Hilbert transform pairs of wavelet bases,” vol. 8, no.
6, pp. 170–173, June 2001.
[21] S. Kyochi, T. Uto, and M. Ikehara, “Dual-tree complex wavelet
transform arising from cosine-sine modulated filter banks,” in Proc.
of ISCAS’09, Taipei, Taiwan, May 2009, pp. 2189–2192.
[22] S. Kyochi and M. Ikehara, “A class of near shift-invariant andorientation-selective transform based on delay-less oversampled evenstacked cosine-modulated filter banks,” IEICE Trans. Fundamentals.,
vol. E93-A, no. 4, pp. 724–733, Apr. 2010.
[23] L. Liang and H. Liu, “Dual-tree cosine-modulated filter bank with linearphase individual filters: An alternative shift-invariant and directional
selective transform,” IEEE Trans. Image Process., vol. 22, no. 12, pp.
5168–5180, Dec. 2013.
[24] R. Ogawa, S. Kyochi, and T. Suzuki, “Design of oversampled cosinesine modulated filter banks for directional image representation,” inProc. of APSIPA ASC’15, Hong Kong, Dec. 2015, pp. 370–373.
[25] H. S. Malvar, “A modulated complex lapped transform and its applications to audio processing,” in Proc. of ICASSP’99, Phoenix, AZ, Mar.
1999, pp. 1421–1424.
[26] M. J. T. Smith and S. L. Eddins, “Analysis/synthesis techniques for
subband image coding,” IEEE Trans. Signal Process., vol. 38, no. 8,
pp. 1446–1456, Aug. 1990.
[27] Y. Tanaka, A. Ochi, and M. Ikehara, “A non-expansive convolution
for nonlinear-phase paraunitary filter banks and its application to image
coding,” in Proc. of ACSSC’05, Pacific Grove, CA, Oct. 2005, pp. 54–
58.
[28] T. Uto, T. Oka, and M. Ikehara, “M -channel nonlinear phase filter banks
in image compression: Structure, design, and signal extension,” IEEE
Trans. Signal Process., vol. 55, no. 4, pp. 1339–1351, Apr. 2007.
[29] T. Suzuki and M. Ikehara, “Reversible symmetric non-expansive
convolution: An effective image boundary processing for M -channel
lifting-based linear-phase filter banks,” IEEE Trans. Image Process.,
vol. 23, no. 6, pp. 2744–2749, June 2014.
[30] S. Kyochi, T. Suzuki, and Y. Tanaka, “A directional and shift-invariant
transform based on M -channel rational-valued cosine-sine modulated
filter banks,” in Proc. of APSIPA ASC’12, Hollywood, CA, Dec. 2012,
pp. 1–4.
[31] R. Ishibashi, T. Suzuki, S. Kyochi, and H. Kudo, “Image boundary
extension with mean values for cosine-sine modulated filter banks,” inProc. of ISCIT’15, Nara, Japan, Oct. 2015, pp. 69–72.
[32] R. D. Koilpillai and P. P. Vaidyanathan, “Cosine-modulated FIR filterbanks satisfying perfect reconstruction,” IEEE Trans. Signal Process.,
vol. 40, no. 4, pp. 770–783, Apr. 1992.
[33] H. S. Malvar, Signal Processing with Lapped Transforms, Norwood,
MA: Artech House, 1992.
[34] “The USC-SIPI Image Database,”
Univ. Southern California,
Signal and Image Processing Institute [Online],
Available:
http://sipi.usc.edu/database/.

11[35] P. P. Vaidyanathan, Multirate Systems and Filter Banks, Englewood
Cliffs, NJ: Prentice Hall, 1992.
[36] S. Mallat, A Wavelet Tour of Signal Processing: The Sparse Way,
Academic Press, 3rd edition, 2008.
[37] D. L. Donoho, “De-noising by soft-thresholding,” IEEE Trans. Inform.
Theory, vol. 41, no. 5, pp. 613–621, May 1995.

Seisuke Kyochi (S’08-M’10) received his B.S. inmathematics from Rikkyo University, Japan, in 2005
and his M.E. and Ph.D. from Keio University, Japan,
in 2007 and 2010. He was a researcher at NTT
Cyberspace Laboratories from 2010 to 2012. In
2012, he joined The University of Kitakyushu, andsince 2015 he has been an Associate Professorof the Faculty of Environmental Engineering. His
research interests include the theory and design ofwavelets/filter banks for efficient image processing
applications.

Ryoma Ishibashi graduated from the Advanced
Production Systems Engineering Course, Gunma
National College of Technology, Japan, and received
the B.E. degree from the National Institution for
Academic Degrees and University Evaluation, Japan,
in 2014. He received the M.E. degree from the
University of Tsukuba, Japan, in 2016. He joined
NEC Corporation in 2016. His current researchinterests are signal/image processing and satellite
communications.
Hiroyuki Kudo (M’88) received the B.Sc. degree from the Department
of Electrical Communications, Tohoku University, Japan, in 1985, and the
Ph.D. degree from the Graduate School of Engineering, Tohoku University,
in 1990. In 1992, he joined the University of Tsukuba, Japan, as an AssistantProfessor. He is currently a Professor with the Division of Information
Engineering, Faculty of Engineering, Information and Systems, Universityof Tsukuba, Japan. He is involved in tomographic image reconstruction for
X-ray CT, PET, SPECT, and electron tomography. His research interests
include medical imaging, image processing, and inverse problems. He is amember of the Japanese Society of Medical Imaging Technology and IEICE,Japan. He received the best paper award several times from various domestic
and international academic societies. In particular, his papers on interior
tomography published in 2008 were selected as Highlights of the two journals,
Physics in Medicine and Biology and Inverse Problems. Since 2011, he has
been Editor-in-Chief of the Journal of Medical Imaging Technology at MIT.

Taizo Suzuki (S’08-M’11-SM’17) received the
B.E., M.E., and Ph.D. degrees in electrical engineering from Keio University, Japan, in 2004, 2006
and 2010, respectively. From 2006 to 2008, he waswith Toppan Printing Co., Ltd., Japan. From 2008
to 2011, he was a Research Associate of the Global
Center of Excellence (G-COE) at Keio University,
Japan. From 2010 to 2011, he was a Research
Fellow of the Japan Society for the Promotion of
Science (JSPS) and a Visiting Scholar at the Video
Processing Group, the University of California, San
Diego, CA. From 2011 to 2012, he was an Assistant Professor in the
Department of Electrical and Electronic Engineering, College of Engineering,
Nihon University, Japan. Since 2012, he has been an Assistant Professor
in the Faculty of Engineering, Information and Systems, University ofTsukuba, Japan. His current research interests are image and video processing,source coding, and multidimensional transforms. Since 2017, he has been an
Associate Editor of IEICE Trans. Fundamentals.