LDMIC: Learning-based Distributed Multi-view Image Coding
Authors: Xinjie Zhang
Jiawei Shao
Jun Zhang
Venue: ICLR 2023 poster (2023)
Keywords: Deep multi-view image compression
distributed source coding
cross-attention mechanism

Abstract: Multi-view image compression plays a critical role in 3D-related applications. Existing methods adopt a predictive coding architecture, which requires joint encoding to compress the corresponding disparity as well as residual information. This demands collaboration among cameras and enforces the epipolar geometric constraint between different views, which makes it challenging to deploy these methods in distributed camera systems with randomly overlapping fields of view. Meanwhile, distributed source coding theory indicates that efficient data compression of correlated sources can be achieved by independent encoding and joint decoding, which motivates us to design a learning-based distributed multi-view image coding (LDMIC) framework. With independent encoders, LDMIC introduces a simple yet effective joint context transfer module based on the cross-attention mechanism at the decoder to effectively capture the global inter-view correlations, which is insensitive to the geometric relationships between images. Experimental results show that LDMIC significantly outperforms both traditional and learning-based MIC methods while enjoying fast encoding speed. Code is released at https://github.com/Xinjie-Q/LDMIC.

Published as a conference paper at ICLR 2023
LDMIC: LEARNING-BASED DISTRIBUTED MULTI- VIEW IMAGE CODING
Xinjie Zhang, Jiawei Shao, Jun Zhang The Hong Kong University of Science and Technology. Hong Kong, China {xinjie fijiee.. zhang, j jiawei. ssao) (atao)ecconnett ust.hk,
hk, ee jejzhangeust. hk
ABSTRACT
Multi-view image compression plays critical role in 3D-related applications. Existing methods adopt a predictive coding architecture, which requires joint en- coding to compress the corresponding disparity as well as residual information. This demands collaboration among cameras and enforces the epipolar geometric constraint between different views, which makes challenging to deploy these methods in distributed camera systems with randomly overlapping fields of view. Meanwhile, distributed source coding theory indicates that efficient data pression of correlated sources can be achieved by independent encoding and joint decoding, which motivates image coding (LDMIC) framework. With independent encoders, LDMIC intro- duces simple yet effective joint context transfer module based on the cross- attention mechanism at the decoder to effectively capture the global inter-view correlations, which is insensitive to the geometric relationships between images. Experimental results show that LDMIC significantly outperforms both traditional and tearning-basee based MIC methods while enjoying fast encoding speed. Code is released at
design a learning-based distributed multi-view
INTRODUCTION
Multi-view image coding (MIC) aims to jointly compress set of correlated images captured from different viewpoints, which is promising to achieve high coding efficiency for the whole image set by exploiting inter-image correlation. It plays an important role in many applications, such as autonomous driving (Yin et al. 2020), virtual reality (Fehn) 2004). and robot navigation (Sanchez- Rodriguez & Aceves-Lopez, 2018). As shown in Figure (a) existing multi-view coding standards, e.g., H.264-based MVC (Vetro et al. 201I) and H.265-based MV-HEVC (Tech et al. 2015), adopt joint coding architecture to compress different views. Specifically, they follow the predictive com- pression procedure of video standards, in which a selected base view is compressed by single image coding. When compressing the dependent view, both the disparity estimation and compensation employed at the encoder to generate the predicted image. Then the disparity information as well as residual errors between the input and predicted image compressed and passed to the decoder. In this way. the inner relationship between different views decreases in sequel. These methods depend on hand-crafted modules, which prevents the whole compression system from enjoying the benefits of end-to-end optimization.
Inspired by the great success of learning-based single image compression (Ballé et 2017 2018
Minnen et al.) 2018 Cheng et al. 2020), several recent works have investigated the application of deep learning techniques stereo image coding, special case of MIC. In particular, [Liu et al. 2019 Deng et al. (2021) Wödlinge et al. (202)), mimicking traditional MIC techniques, adopt unidirectional coding mechanism and explicitly utilize the disparity compensation prediction the pixel/feature space to reduce the inter-view redundancy. Mcanwhile, Lei et al. (2022) introduces bi-directional coding framework, called as BCSIC, to jointly compress left and right images simul- taneously for exploring the content dependency between the stereo pair. These rudimentary studies demonstrate the potentials of deep neural networks (DNNs) in saving significant bit-rate for MIC.
However, there several significant shortcomings hampering the deployment and application scope of existing MIC methods. Firstly, both the traditional and learning- based approaches demand inter-view prediction at the encoder, iee, joint encoding, which requires the cameras to communi-

---

Published as a conference paper at ICLR 2023
Jain Eneuods
Juhn Doodle
Enoole
Enodle
Figure 1: Overview of different multi-view image coding architectures, including (a) joint encod-
ing architecture and (b) the proposed symmetric distributed coding architecture.
cate with each other to transmit the data to an intermediate common receiver, thereby consuming a tremendous amount of communication resources and increasing the deployment cost (Gehrig & Dragotti 2007). This is undesirable in applications relevant to wireless multimedia sensor networks (Akyildiz et al. 2007). An alternative is to deploy special sensors like stereo cameras as the encoder devices to acquire the data, but these devices are generally more expensive than monocular sensors and suffer from limited field of view (FoV) due to the constraints of distance and position between built-in sensors (Li] 2008). Secondly, most of the prevailing schemes, except BCSIC, are developed based on disparity correlations defined by the epipolar geometric constraint (Scharstein & Szeliski, 2002]. which usually requires to know the internal and external parameters the camera in advance, such as camera locations, orientations, and camera matrices. Whereas, it is difficult for a distributed camera system without communication to access the prior knowledge of cameras (Devarajan et al.) 2008). For example, the specific location information of cameras in autonomous driving is usually not expected to be perceived by other vehicles or infrastructure in order avoid leaking the location and trajectory of individuals (Xiong et al.) 2020). Finally, shown in Table I and Figure 4 com- pared with state-of-the-ar (SOTA) learning-bas based single image codecs (Minnen et al. 2018] Cheng et al. 2020), existing DNN-based MIC methods are not competitive in terms of rate-distortion (RD) performance, which is potentially caused by inefficient inter-view prediction networks.
To address the above challenges, resort to innovations in the image coding architecture. Partic- ularly, our inspiration comes from the Slepian- Wolf (SW) theorem (Slepian & Wolf 1973] Wolf 1973] on distributed source coding (DSC) [ The SW theorem illustrates that separate encoding and joint decoding of two or more correlated sources can theoretically achieve the same compres- sion rate as joint encoding- -dcooding scheme under lossless compression. It has been extended to the lossy case by Berger (1978) and Tung (1978), which provides the inner and outer bounds of the achievable rate region. Based on these information-thes theorretc results on DSC, we develop a dearning-base distributed multi-view image coding (LDMIC) framework. Specifically, to avoid collaboration between different cameras, as shown in Figure [(b) each view image is mapped to the corresponding quantized latent representation by an individual encoder, while a joint decoder is used to reconstruct the whole image set, which can successfully avoid the communication among cameras the usage of special sensors. This architectural innovation is theoretically supported by the DSC theory. Instead of disparity-based correlations, we design joint context transfer (JCT) module based on the cross-attention mechanism agnostic geometry priors to exploit the global content dependencies between different views at the decoder, making our approach applicable to arbitrary multi-camera systems with overlapping FoV. Finally, since the separate encoding and joint decoding scheme is implemented by DNNs, the end-to-end RD optimization strategy is leveraged to implicitly help the encoder to learn remove the partial inter-view redundancy. thus improving the compression performance of the overall system. In summary, our main contributions are as follows:
To the best of our knowledge, this is the first work to develop a novel deep learning-based view-symmetric framework for multi-view image coding. decouples the inter-view oper- ations at the encoder, which is highly desirable for distributed camera systems.
We present a joint context transfer module at the decoder to explicitly capture inter-view correlations for generating more informative representations. We also propose end-to- end encoder-decoder training strategy to implicitly make the latent representations more compact.
Mooe details about the theorem and proposition of distributed source coding are provided in Appendix 6.4
2

---

Published as a conference paper at ICLR 2023
Extensive experimental results show that our proposed framework is the first distributed codec achieving comparable coding performance to the SOTA joint encoding-decoding schemes, implying the effectiveness of the inter-view cross-attention mechanism com- pared with the conventional disparity-based prediction. Moreover, our proposed frame- work outperforms the commmetric-hssett coding framework NDIC (Mital et al.] 2022b), which demonstrates the advantage of the view-symmetrie design over the asymmetric one.
RELATED WORKS
Single Image Coding. In the past decades, various standard image codecs have been developed, including JPEG (Wallace) 1992). JPEG2000 (Skodras et al. 200T), BPG (Bellard 2014), and VVC intra (Bross et al.) 2021). They generally apply three key ideas to reduce redundancy: (i) trans- form coding, e.g., discrete cosine transform, to decrease the spatial correlation, (ii) quantization of transform coefficients to filter the irrelevancy related to the human visual system, and (iii) entropy coding lessen the statistical correlation of the coded symbols. Unfortunately, these components are separately optimized, making it hard to achieve optimal coding efficiency.
Recently, end-to-end image compression has engaged increasing interests, which is built upon the transform coding paradigm with nonlinear transform and powerful entropy models for higher com- pression efficiency. Nonlinear transform is used to produce compact representations, such as gen- eralized divisive normalization (GDN) (Ballé et al.] 2013), the self-attention block (Cheng et al.) 2020), wavelet-like invertible transform (Ma et al., 2020) and stacks of residual bottleneck blocks (He et al. 2022). To approximate the distribution of latent representations, many advanced entropy models have been proposed. For example, [Blléé et al. (2017: 2018) put forward the factorized and hyper prior entropy models for the first time. Then the regroseeresssvve context model (Minnen al., 2018) is combined into the hyper prior to effectively reduce the spatial redundancy of images at the expense of high decoding latency. In order improve the decoding speed, Minnen & Singh (2020) and He et al. (202T) investigate the channel-wise and spatial-wise context versions, respectively. These existing works are considered as important building blocks for our scheme.
Multi-view Image Coding, Conventional MIC standards (Vetro et al. 2011] Tech et al. 2015) are derived from key frame compression methods designed for multi-view video codecs. Since these methods are still in the development stage and only support YUV420 format, they are uncompetitive against single image codecs that allow the YUV444 or RGB format. Mcanwhile, existing learning- based MIC approaches (L.iu et al. 2019 Deng et al. 2021 Wödlinger et al., 2022 Lei et al. 2022) mainly focus on stereo images, and it is difficult to effectively extend them the general multi-view scenario. Moreover, they can only handle fixed number of views. In contrast, our framework exerts average pooling to merge the information between multiple views, making it insensitive to the number of viewpoints.
Distributed Source Coding. There have been some works developing multi-view compression methods based on DSC. They are typically built on the setting of coding with side information (Zhu et al.] 2003: Thirumalai et al.) [2077] Chen et al., 2008: Wang et al.) 2012). where one view selected reference and compressed independently. For other views, the joint decoder uses the reference as side information to capture the inter-view correlations to reduce the coding rate. Recent learning-based distributed multi-viww image compression concentrates on this asymmetric paradigm (Ayzik & Avidan 2020 Whang et al.] 2021; Wang et al. 2022; Mital et al. 2022a b). Nevertheless, this architecture suffers from high transmission cost for the primary sensor, since requires a hierarchical relationship between different cameras, leading to the unbalanced coding rates among them (Tosic & Frossard, (0099).
Different from the above works, we consider more practical symmetric coding pattern illustrated in Figure ((b). where all cameras are treated as equal status. While traditional symmetric coding schemes TTiiuualaai et al. 2008 Gehrig & Dragotti 2009) utilize sispaiity-bssed estimation at the decoder to reduce the transmission cost, we get rid of the disparity compensation prediction and adopt the cross-attention mechanism (Vaswani et al.) 2017) to capture the global relevance be- tween different views, which effectively improves the compression performance and broadens the application scope. As far knowledge, our study is the first in applying DNNs into symmet- ric distributed coding and achieving the RD performance comparable to joint decodddng schemes.
3

---

Published as a conference paper at ICLR 2023
Lanky RALL
X1
GBN
Leily SeLU
Cand
Tankyy SALL
9,
Figure 2: The proposed LDMIC framework with an auto-regressive entropy model, where (KY(A) and [k) represent the set of all the view features except for the k-th view feature ÿr. and hxs re- spectively. Convolution/dccoovolutiin parameters are formatted as (the number of output channels, kernel size, stride). denotes quantization. AE and AD represent arithmetic encoder and decoder, respectively.
PROPOSED METHOD
3.1
THE OVERALL ARCHITECTURE OF LDMIC
Figure 2 depicts the network architecture of the proposed method. Let = (1, K} denote the image index set. Given group of multi-view images dR {æ), .2...... xxx each image ay is independently mapped to the corresponding representation y/s by the encoder Fg with shared network parameters. Then yı is quantized to ÿr. After receiving all the quantized representations gg. the joint decoder JD exploits the inter-view correlations among 9K to reconstruct the whole image set ēg. The compression procedure is described
yk (T Φ), 4k E K,
ÿı Qy:,, vk E K
Øx JD((x:)) 0),
where Φ and 0 are optimized parameters of the encoder and decoder. Since the quantizer Q not differentiable, we apply the mixed quantization approach proposed in Minnen & Singh (2020) during training. Specifically, the latent representation yı: with an additive uniform noise is taken as the input to the entropy model for estimating the bitrate, while the rounded representation with (STE) gradient flows to the joint decoder for reconstruction.
thaighh-trrough-estiattiee
To apply entropy coding to reduce the statistical correlation of the quantized representation ÿr. each element i is modelled as a univariate Gaussian random variable with its mean Pk,4 and standard deviation σki by introducing side information Êc,ú, where denotes the position of each element vector-vaie valued signal. The probability distribution Pax (2, of ÿt is expressed as follows:
Po.le (#k|&s) ( o
Meanwhile, context model also combined with the entropy model for effectively reducing the spatial redundancy of latent ÿt. The selection of the context model depends on the specific needs of different applications. We choose an auto-regressive model (Minnen et al. (018)) and a checkerboard model (He et al. 2021) for better coding efficiency and faster coding speed, respectively.
3.2 JOINT CONTEXT TRANSFER MODULE
Due to the overlap between the cameras' FoV, there exist significant inter-view correlations in the feature space, which inspires property for generating more informative representations. As shown in Figure 3 the proposed JCT module receives multi-view features fx as inputs, learns an inter-view context for each view feature, and refines the input features based on the corresponding inter-view contexts. Note that there are K parallel paths in the JCT module. Each path shares the same network parameters and follows three-step process described below to obtain the refined representations ft.
propose a joint context transfer (JCT) module exploit this
Feature extraction. We firstly utilize two residual blocks to extract the representative feature f from the k-th view ft. Each residual block, as depicted in Figure is composed of two consecutive convolution layers with Leaky ReL.U activation functions.
4

---

Published as a conference paper at ICLR 2023
fk
Residuel Black
Residusi Block
Mulli-Seed Cross Attention
Residual Block 2
Residual Block
J+
f
Soky 3660
Takdy Bou
Average Pooling
Figure 3: Illustration of the k-th path in the proposed joint context transfer module, where (K((A) denotes the set of all the view representations except for the current view representation f.-
Multi-view fusion. All the representations fK from the feature extraction module except fg are aggregated to preliminary context f/ via simple average pooling the dimension of the number the input features:
f K
Σ f. (EKYE)
K. By this aggregation operation, we achieve fu- sion between any number of view features. In addition, is observed that more complex pooling approaches can be developed to further improve the performance.
where K{[k}
After getting the aggregated context, we apply multi-head cross- coss--tteention module to exploit the dependency between fa and f.. Since the original attention module incurs high memory and com- putational cost under a large spatial dimension of input, we adopt the resource resource-efficien efficient attention convolution layer and a reshape operation to transform f& E Hx and fx E R i.e., query Qs = Conv(fa) E Tgxxkxd, key K Conv(fe) E Rxxxxxd and value V4 = Conv (fx) E Rnxxhxd where n H x W and denotes the number of heads. The notations d, d1 and d2 the channel dimensions of input, key (query) and value in a head, respectively. Then the multi-head cross-attentie module is applied as:
[Shen et al. (202T). Specifically, we use
A row (o (K.)) VV..). Vi = 1, ,,
4)
(1)- Conv(A .... $ Ak,R),
where Trow (Geal) denotes applying the softmax function along each row (column) of the matrix, and b is the channel-wind concatenation. The context fxx fxx(1)->e relevant the k-th view feature is extracted and will be injected into the current feature the next step.
Refinement. Based on the learned inter-view context more informative feature f::
( the input feature fa is refined to
+
f().
where F(-) consists of two consecutive residual blocks. As shown Figure 2 the JCT module is placed before the first and third deconvolution layers to connect the different- view decoding stream for feature aggregation and transformation.
3.3 TRAINING
The target of LDMIC is optimize the trade-off between the number of encoded bits and the reconstruction quality. Therefore, a training loss composed two metrics used:
K L -= d(zr xx) + Σ (R(ga) + R(2x))
K
(6)
where d(xxk, tx)) is the distortion between Ty and E under a given metric, such as mean squared error (MSE) R(ğx) and R 2x) represent the estimated compression rates of the latent representation ÿr and the corresponding hyper representation ε,, respectively. À is a hyper parameter that controls the trade-off between the bit rate cost R and distortion D.

---

Published as a conference paper at ICLR 2023
100- (181
POOH 0882
pOUR
04
045
025
90
V15
Dudinal
00
6.3
6.
(d)
(f)
Figure 4: Rate-distortion curves of our proposed methods compared against various competitive baselines.
EXPERIMENTS
4.1 EXPERIMENTAL SETUP
Datasets. To compare with the recently developed learning-based stereo image compression meth- ods, two common stereo image datasets, i.e., Instereo2K (Bao et al. (2000) and Cityscapes (Cordts iet al. 2016), are chosen to evaluate the coding efficiency of the proposed framework. Apart from testing stereo image datasets related 3D scenes, we also select pedestrian surveillance dataset, i.e., WildTrack (Chavdarova et al., (2018), acquired by seven random placed cameras with overlap- ping FoV, which is to demonstrate the potentials of our proposed framework in distributed camera systems without epipolar geometry relationship between images. More details about datasets are provided in Appendix 6..5
Benchmarks. The competing baselines can be split into three categories: (1) Separate model in- dependently compresses each image, whose typical SOTA representatives are BPG (Bellard 2014), VVC-intra (Bross et al. 2021), Minnen et al. 2018) and Cheng et al. (220)) For BPG and VVC- intra, we disable chroma subsampling. (2) Joins model has access to a set of multi-view images and explicitly utilizes the inter-viet redundancy to achieve high compression ratio. According to per- formance comparisons in Wödlinger et al. (2022). conventional video standards can be applied in the MIC, where each set of multi-view images is compressed
multi-frame video sequence by using both HEVC (Sullivan et al., 2012) and VVC (Bross et al. (202I) with lowdelay_P configuration as well as YUV444 input format. We also test MV-HEVC (Tech al. 2015) with the multi-view intra mode. Apart from that, we report the results of several recent
stereo image codecs on the InStereo2K and Cityscapes datasets, including DSIC (Liu al. 2019), two variants of HESIC (Deng et al. 2021), BCSIC (Lei et al.] (002). and SASIC (Wödlinger et al. (022). (3) Distributed model only uses the joint decoder to implicitly reduce the inter-view dependency. We compare our
method with NDIC based asymmetric DSC (Mital et al.) 2022b) to demonstrate the superiority of symmetric DSC. More details on bascline settings are given in Appendix 6.5]
Metrics. The distortion between the reconstructed and original images is measured by peak signal- to-noise ratio (PSNR) and multi-scale structural similarity index (MS-SSIM) (Wang et al. 2003).
Besides assessing RD curves, compute the Bjøntegaard Delta bitrate (BDBR) (Bjontegaard 2001) results to represent the average bitrate savings at the same distortion level.
Implementation Details. We train models with five different À values, where X 256, 512, 1024, 2048, 4096 (8, 16, 32, 64, 128) under MSE (MS-SSIM) For MSE optimized mod- els, they are trained from scratch for 400 epochs InStereo2K/Citssaapes and 700 epochs on

---

Published as a conference paper at ICLR 2023
Table 1: Comparison of BDBR relative BPG different datasets, with the best results in
Bold and second-best ones in underlined.
InStereo2K
Cityscapes
WildTrack (CI, C4)
Categories
Methods
PSNR
MS-SSIM
PSNR
MS-SSIM
PSNR
MS-SSIM
Minnen2018
-7.44%
-34.37%
-21.8%
-46.74%
-00440%
-47.73%
Separate
Cheng2020
-19.71%
-41.5%
-27.86%
-49.63%
19.23%%
-52.54%
VVC-intra
-3.23%
-17.38%
-4.14%
-22.12%
-10.76%
-24.8%%
VVC
-30.5%%
-3.880%
-2.85%%
46.35%
-4.18%
99.3%%
HEVC
-15.54%
-14.70%
-23.40%
-24.48%
39.04%
23.27%
MV-HEVC
2.83%
-4.75%
-18.57%
-0.17%
33.88%
9.35%
HESIC
0.47%
-39.55%
-7.92%
-45.14%
Joint
HESIC+
-15.06%
-3.56%
-21.70%
-51.33%
DSIC
107.88%
40.04%
-188%
-38.26%
BCSIC
23.80%
-56.11%
SASIC
-19.83%
-23.0%%
-0033%%
-30. 10%
NDIC Proposed-fas fast Proposed
13.98%
-34.38%
7.36%
-38.07%
3.94%
-51.08%
Distributed
-29.68%
-99.9%%
-28.30%
-53.61%
-26.69%
-55.08%
-41.69%
-59.20%
-40.14%
-62.12%
-31.21% %
-67.77%
Table 2: Complexity of learning-basee image codees evaluated pair of stereo images with the resolution as 832x 1024 in the InStereo2K dataset, where the encoding latency of DSC-based schemes is determined by the maximum time for independent encoding of each image.
Encoder
Decoder
Methods
FLOPs
Params
Time
FLOPs
Params
Time
DSIC
2415.29G
79.26M
25.97s
3378.65G
75.78M
26.45s
HESIC
285.3G
32.08M
3.23s
1197.22G
29.55M
16.15s
HESIC+
205.71G
17.02M
16.79s
1122.87G
15.28M
49.96s
SASIC
531.42G
3.58M
10.66s
2532.87G
4.48M
34.45s
NDIC
163.93Gx2
7.25Mx2
3.19s
1245.89G
25.04M
9.93s
Proposed-fai
194.15Gx2
11.24Mx2
2.37s
1851.96G
15.24M
11.48s
Proposed 187.39Gx2
11.24Mx2
9.44s
1838.42G
15.24M
47.87s
WildTrack by using Adam optimizer (Kingma & Ba 2014), in which the batch size is taken 8.
The learning rate is initially set as 10 and decreased by a factor of 2 every 100 epochs until it reaches 400 epochs. As for MS-SSIM optimized models, we fine-tune the MSE optimized networks for 300 (400) epochs with the initial learning as 5 x 10-5 stereo anulti-camera) image dataset. During training, each image randomly flipped and cropped to the size of 256 256 for data aug- mentation. The whole framework is implemented by CompressAl (Bégaint et al. 2020) and trained on a machine with NVIDIA RTX 3090 GPU.
4.2 EXPERIMENTAL RESULTS
Coding performance. Figure 4 presents the RD curves of all compared methods and Table gives the corresponding BDBR results of each codec relative to BPG. For InStereo2K and Cityscapes, the proposed method outperforms most of these compression baselines both PSNR and MS-SSIM. which implies that relying only on joint decoding can effectively reduce the inter-view redundancy between different views. For example, when compared with Cheng2020 (SASIC), our method and the fast variant reduce around 21 1,98% % (21.88 1.6%%) and 9.97% (9.85%) bits in terms of PSNR, respec- tively. Since stereo images contain plenty of homogeneous regions suited for traditional coding, VVC achieves up to 30.54% and 52.85% compression efficiency when measured by PSNR, but no- tice that requires joint encoding. On the InStereo2K dataset, VVC underperforms our method by margin with about 0.44dB coding gains in PSNR due larger variation in image content. In addition, our proposed framework attains better reconstruction quality measured by MS-SSIM at the same bitrate when compared with VVC.
As seen from Figure 4(c) and 4()) we select the images acquired by two cameras, Cl and C4, on the WildTrack dataset evaluate the compression performance of different methods without

---

Published as a conference paper at ICLR 2023
InStereo2K Dataset
PSNR (48)
Bid socee
GMl
bits
Figure 5: Ablation study. Joint Enc-Dec and Sep Enc-Dec denote inserting and removing the JCT module at the encoder and decoder, respectively. Concatenation. SAM and BiCTM represent differ- ent inter-view operations to replace the proposed JCT module at the decoder. W/O Joint Training is fix the pretrained encoder including the entropy model and only train the joint decoder.
Left Image
Hight Image
represenaatioo Tratntng
reperretacoon
repremmtaaion
Figure 6: Visual examples from the InStereo2K dataset, where assemble all channels of the latent representation Q - [4x) to display the feature map.
using additional information from other cameras. It is observed that the traditional video codecs perform worse than the corresponding intra-frame ones due to lots of heterogeneous overlapping regions, which makes difficult for standard video codees to effectively capture the inter-view redundancy by using compensation-based predictions. However, our proposed framework relies on the cross-attention mechanism to exploit the correlations of different views from the perspective of global receptive fields, thereby providing up to 31.21 % and 67.77% bitrate saving in PSNR and MS- SSIM, respectively. The remarkable results demonstrate that the proposed LDMIC framework is promising solution to meet the compression needs of distributed camera systems. The RD curves on the multi-camera case have similar trends with that on the two-camera one, which are provided in Appendix
Moreover, compared with asymmetric DSC-based NDIC, the proposed method saves 55.67%, 47.5% and 35.15% bits in PSNR three datasets (InStereo2K, Cityscapes, Wild'Track). For the proposed-fast variant with the checkerboard entropy model, the improvements are also adequate, i.e. 43.66%, 35.66% and 30.63%. This set of results indicate that the usage of bi-directional in- formation based on symmetric DSC can better exploit the inter-view correlations to bring higher coding gains. Additionally, our methods have better compression efficiency MS-SSIM than in PSNR, which is partly caused by exploiting the inter-view correlations in the feature space rather than pixel space at the decoder. Thus, the network tends focus on structure information instead of pixel information.
Computational complexity. Table shows the computational complexity of seven image codecs running on an Intel Xeon Gold 6230R processor with base frequency 2. 2.10GHz and single CPU core, including the number of FLOPs, the model parameters and the coding latency. Different from the joint models, methods designed on DSC decouples the inter-view operations at the encoder, which allows image-level parallel processing. Therefore, the proposed-fat variant enjoys about 1.36 10.95 and 1.41 4.35 times encoding and decoding speedup against the learned joint schemes (i.e., DSIC, HESIC, HESIC+, SASIC). Even the auto-regressive entropy model is used, the encoding of method is still faster than that of both DSIC and SASIC based hyper
8

---

Published as a conference paper at ICLR 2023
Table 3: Bitrate savings for two-view images with cameras C1 and C2 the number of viewpoints increases on the WildTrack dataset, where the case of K = 2 is set as the anchor.
Number of cameras
= K 3 K 4 K
K-6 K =
Bitrate saving (%)
0.0053 0.0801
1.0919
1.4161
1.5004
prior. Moreover, our proposed fast variant with better coding efficiency achieves similar coding time with another DSC-based method NDIC, which demonstrates the superiority of symmetric DSC in coding speed and compression efficiency. For more details on comparison between our methods and traditional codecs, please refer to Appendix 6.2
4.3 ABLATION STUDY
Inter-view Fusion. To verify the contribution of the JCT module for fusing inter-view information, set of ablation experiments are conducted on the InStereo2K dataset with RD curves shown in Figure SS Specifically, we allow (forbid) both the encoder and the decoder to access the inter- view context, which provides an upper (lower) bound on the performance of the proposed method and is denoted by Joint (Sep) Enc-Dec. In this case, the PSNR with (without) the JCT module the encoder (decoder) improves (drops) by about 0. 0.16dB (0.73dB) at the same hpp level. We further report the compression results when the JCT module is directly replaced by other inter-view fusion operations such as concatenation in Mital et al. (2022b), stereo attention module (SAM) in WWödlinger et al. (2022) and bi-directional contextual transform module (Bi-CTM) in Lei et al. (2022). These operations lead to an increase of the bitrate by 32.73%, 27.99%, 10.1 1% compared with method. The experimental results indicate that our proposed JCT module have powerful capability in capturing inter-view correlations and generating more informative representations.
Joint Training Strategy. In this paper, we exploit the benefit of joint training to implicitly help the encoder to learn removing the partial inter-view redundancy. Thus, the latent representation is expected to be more compact. To investigate its effect, perform experiment by only training the joint decoder with the fixed pre-trained encoder and entropy model. As shown in Figure 3 our approach outperforms the W/O Joint Training method by 0.225 dB. In Figure 6 we provide further visual comparisons. It is noted that the latent feature maps with joint training strategy contain more elements with low magnitudes, which requires much fewer bits for encoding.
Number of views. Table shows the impact of different numbers of views on coding efficiency. We compare the bitrate of cameras C1 and C2 when incorporating different numbers of views during decoding. The bitrate saving increases gradually as more information is received from different cameras. Due to only using a simple average pooling to merge multi-view information to the inter- view context, we get marginal coding gains when incorporating more views. is possible to further improve the compression gains of our framework by using more complex aggregation approaches.
5 DISCUSSION
In this paper, we presented a novel end-to-end distributed multi-view image coding framework nicknamed LDMIC. Our proposal inherits the advantages of traditional distributed compression in image-level parallelization, which is desirable for distributed camera systems. Meanwhile, leverag- ing the insensitivity of the cross-attention mechanism to epiploar geometric relations, develop joint context transfer module to account for global correlations between images from different viewpoints. Experimental results demonstrate the competence of LDMIC in achieving higher cod- ing gains than existing leasning-based joint and separate ricoding-decoding schemes. Moreover, compared with learned joint models, the LDMIC fast variant enjoys a much lower coding complex- ity with on-par compression performance. To the best of our knowledge, this is the first successful attempt of the distributed coding architecture to fight against the performance of the joint coding paradigm under the lossy compression case.
Based on the proposed framework, there are two clear directions to be explored in the future. On one hand, mentioned in Section 4 it is interesting to investigate how to more effectively incorporate different view information to generate a better inter-view context. One the other hand, it is worth exploring how to extend the framework multi-view video compression.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGMENTS
This work was supported by the NSFC/RGC Collaborative Research Scheme (Project No. CRS HKUST603/22).
REFERENCES
lan Akyildiz, Tommaso Melodia, and Kaushik R Chowdhury. A survey on wireless multimedia sensor networks. Computer networks, 51(4)9221-66, 2007.
Sharon Ayzik and Shai Avidan. Deep image compression using decoder side information. In Euro- pean Conference on Computer Vision, pp. 699-714. Springer, 2020.
Johannes Ballé, Valero Laparra, and Eero Simoncelli. Density modeling of images using a gener- alized normalization transformation. arXiv preprint arXiv:1511.0628/ 2015.
Johannes Ballé, Valero Laparra, and Eero P Simoncelli. End-to-end optimized image compression. In International Conference on Learning Representations, 2017.
Johannes Ballé, David Minnen, Saurabh Singh, Sung Jin Hwang, and Nick Johnston. Variational image compression with a scale hyperprior. In International Conference on Learning Represen- tations, 2018
Wei Bao, Wei Wang, Yuhua Xu, Yulan Guo, Siyu Hong, and Xiaohu Zhang. Instereo2k: A large real dataset for stereo matching in indoor scenes. Science China Information Sciences, 63(11); 1-11, 2020.
Jean Bégaint, Fabien Racapé, Simon Feltman, and Akshay Pushparaja. Compressai: a py- torch library and evaluation platform for end-to-end compression research. arXiv preprint arXiv:2011.03029, 2020.
Fabrice Bellard. Bpg image format. https::/bellarr..cc.m
2014.
Toby Berger. Multiterminal source coding. The Information Theory Approach to Communications (CISM Courses and Lectures), 229:171 129:171-,31, 1978.
Gisle Bjontegaard. Calculation of average psnr differences between rd-curves. ITU-T VCEG-M33, April, 2001, 2001.
Benjamin Bross, Ye-Kui Wang, Yan Ye, Shan Liu, Jianle Chen, Gary Sullivan, and Jens-Rainer Ohm. Overview of the versatile video coding (vve) standard and its applications. IEEE Transac- tions on Circuits and Systems for Video Technology, 31(10))3736-3 3764, 2021.
Tatjana Chavdarova, Pierre Baqué, Stéphane Bouquet, Andrii Maksai, Cijo Jose, Timur Bagautdi- nov, Louis Lettry, Pascal Fua, Luc Van Gool, and François Fleuret. Wildtrack: A multi-camera hd dataset for dense unscripted pedestrian detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5030-5039, 2018
David Chen, David Varodayan, Markus Flierl, and Bernd Girod. Distributed stereo image coding with improved disparity and noise estimation. In 2008 IEEE International Conference on Acous- tics, Speech and Signal Processing, pp. 1137-1140. IEEE, 2008
Zhengxue Cheng, Heming Sun, Masaru Takeuchi, and Jiro Katto. Learned image compression with discretized gaussian mixture likelihoods and attention modules. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7939-7948, 2020.
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3213-3223, 2016.
Xin Deng, Wenzhe Yang. Ren Yang, Mai Xu, Enpeng Liu, Qianhan Feng, and Radu Timofte. Deep homography for efficient stereo image compression. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 1492-1501, 2021.
10

---

Published as a conference paper at ICLR 2023
Dhanya Devarajan, Zhaolin Cheng, and Richard J Radke. Calibrating distributed camera networks. Proceedings of the IEEE, 96(10):1625- 1639, 2008.
Christoph Fehn. Depth-image-base based rendering (dibr), compression, and transmission for a new approach on 3d-Iv. In Stereoscopic displays and virtual reality systems XI, volume 5291, pp- 93-104. SPIE, 2004.
Nicolas Gehrig and Pier Luigi Dragotti. Distributed compression of multi-view images using geometrical coding approach. In 2007 IEEE International Conference on Image Processing, volume 6, pp. VI-421. IEEE, 2007.
Nicolas Gehrig and Pier Luigi Dragotti. Geometry-driven distributed compression the plenoptic function: Performance bounds and constructive algorithms. IEEE transactions on image process- ing, 18(3):557447, 2009.
Dailan He, Yaoyan Zheng, Baocheng Sun, Yan Wang, and Hongwei Qin. Checkerboard context model for efficient learned image compression. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14771-14780, 2021.
Dailan He. Ziming Yang, Weikun Peng, Rui Ma, Hongwei Qin, and Yan Wang. Elie: Efficient learned image compression with unevenly grouped space-channel contextual adaptive coding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp-
5718-57277 2022.
Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
Jianjun Lei, Xiangrui Liu, Bo Peng, Dengchao Jin, Wanqing Li, and Jingxiao Gu. Deep stereo image compression via bi-directional coding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pultern Recognition, pp. 19669-19678, 2022.
Shigang Li. Binocular spherical stereo. IEEE Transactions on intelligent transportation systems, (4):589 00099960022200 2008.
Jerry Liu, Shenlong Wang, and Raquel Urtasun. Dsic:
Deep stereo image compression. Proceed- ings of the IEEE/CVF International Conference on Computer Vision, pp. 3136-3145, 2019.
Haichuan Ma, Dong Liu, Ning Yan, Houqiang Li, and Feng Wu. End-to-end optimized versatile image compression with wavelet-like transform. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020.
David Minnen and Saurabh Singh. Channel-wise autoregressive entropy models for learned image compression. In 2020 IEEE International Conference Image Processing (ICIP), pp. 3339- 3343. IEEE, 2020.
David Minnen, Johannes Ballé, and George D Toderici. Joint autoregressive and hierarchical priors for learned image compression. Advances in neural information processing systems, 31, 2018.
Nitish Mital, Ezgi Ozyilkan, Ali Garjani, and Deniz Gunduz. Neural distributed image compression with cross-attention feature alignment. arXiv preprint (xXiv:2207.08488 2022a.
Nitish Mital, Ezgi Özyılkan, Ali Garjani, and Deniz Gündüz. Neural distributed image compression using common information. In 2022 Data Compression Conference (DCC), pp. 182-191. IEEE, 2022b.
Jose-Pablo Sanchez-Rodrigue and Alejandro Aceves-Lopez. A survey on stereo vision-based au- tonomous navigation for multi-rotor muavs. Robotica, 36(8): 1225-1233, 2018.
Daniel Scharstein and Richard Szeliski. A taxonomy and evaluation of dense two-frame stereo correspondence algorithms. International journal of computer vision. 47(1):7-42, 2002.
SD Servetto. Multiterminal source coding with two encoders-i: a computable outer bound, 2006.
IEEE Transactions Information Theory, 2006.

---

Published as a conference paper at ICLR 2023
Zhuoran Shen, Mingyuan Zhang, Haiyu Zhao, Shuai Yi, and Hongsheng Li. Efficient attention: Attention with linear complexities. In Proceedings of the IEEE/CVF winter conference on appli- cations of computer vision, pp. 3531-3539, 2021.
Athanassios Skodras, Charilaos Christopoulos, and Touradj Ebrahimi. The jpeg 2000 still image compression standard. IEEE Signal processing magazine, 18(5):36-58, 2001.
David Slepian and Jack Wolf. Noiseless coding of correlated information sources. IEEE Transac- tions on information Theory, (9(4):471-48) 1973.
Gary J Sullivan, Jens-Raines Ohm, Woo-Jin Han, and Thomas Wiegand. Overview of the high efficiency video coding (heve) standard. IEEE Transactions on circuits and systems for video technology, 16412):149-1166, 2012.
Gerhard Tech, Ying Chen, Karsten Müller, Jens-Rainer Ohm, Anthony Vetro, and Ye-Kui Wang. Overview of the multiview and 3d extensions of high efficiency video coding, IEEE Transactions on Circuits and Systems for Video Technology, 26(1):35 49, 2015.
Vijayaraghavan Thirumalai, Ivana Tosic, and Pascal Frossard. Distributed coding of multiresolution omnidirectional images. 2007 IEEE international conference on image processing, volume 2, pp. II-345. IEEE, 2007.
Vijayaraghavan Thirumalai, Ivana Tosic, and Pascal Frossard. Symmetric distributed coding of stereo omnidirectional images. Signal Processing: Image Communication, 23(5):379-390, 2008.
Ivana Tosic and Pascal Frossard. Distributed multi-view image coding with learned dictionaries. Technical report, 2009.
Sui-Yin Tung. Multiterminal source coding. Ph. D. dissertation, School of Electrical Engineering, Cornell University, 1978.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural informa- tion processing systems, 30, 2017.
Anthony Vetro, Thomas Wiegand, and Gary J Sullivan. Overview of the stereo and multiview video coding extensions of the 264/mpeg-4 ave standard. Proceedings of the IEEE, 99(4):626-642, 2011.
Gregory Wallace. The jpeg still picture compression standard. IEEE transactions on consumer electronics, 38iixxxvii,xxxiv, 1992.
Jin Wang, Yunhui Shi. Yinsen Xing, Nam Ling, and Baocai Yin. Deep correlated image set com- pression based on distributed source coding and multi-scale fusion. In 2022 Data Compression Conference (DCC), pp. 192-201. IEEE, 2022.
Shuang Wang, Lijuan Cui, Samuel Cheng, Lina Stankovic, and Vladimir Stankovic. Onboard low- complexity compression of solar stereo images. IEEE transactions image processing, 21(6): 3114-3118, 2012.
Zhou Wang, Ecro P Simoncelli, and Alan C Bovik. Multiscale structural similarity for image quality assessment. In The -hreet-Seventh Asilomar Conference on Signals, Systems Computers, 2003, volume 2, pp. 1398-1402. Ieee, 2003.
Jay Whang, Anish Acharya, Hyeji Kim, and Alexandros Dimakis. Neural distributed source coding. arXiv preprint arXiv:2/06.02799 2021.
Matthias Wödlinger, Jan Kotera, Jan Xu, and Robert Sablatnig. Sasic: Stereo image compression with latent shifts and stereo attention. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 661-670, 2022.
Jack Wolf. Data reduction for multiple correlated sources. Colloquium on Microwave Communi- cation, pp. 287-295, 1973.
12

---

Published as a conference paper at ICLR 2023
Zuobin Xiong, Zhipeng Cai, Qilong Han, Arwa Alrawais, and Wei Li. Adgan: Protect your location privacy in camera data of auto-driving vehicles. IEEE Transactions on Industrial Informatics, 17 (9):6200-6211) 2020.
Huan Yin, Yue Wang, Li Tang, Xiaging Ding, Shoudong Huang, and Rong Xiong. 3d lidar map compression for efficient localization on resource constrained vehicles. IEEE Transactions on Intelligent Transportation Systems, 22(2):837-852, 2020.
Xiaoqing Zhu, Anne Aaron, and Bernd Girod. Distributed compression for large camera arrays. In IEEE Workshop Statistical Signal Processing, 2003, pp. 30-33. IEEE, 2003.
13

---

Published as a conference paper at ICLR 2023
APPENDIX
6.1
RD CURVES ON MULTI-CAMERA WILDTRACK DATASET
WieThack Dataeet (C1-C7)
We5STack Butaset (CHC)
24
38
22
PSNR (d8)
36
MS SSIM
18
32
HOW
14
Dutroutoc
IEYC
Saparce
Prosuddoo
8m6
Praoee
Regered
Ki senwro00
0.0
9.3
0.2
0.3
0.4
0.5
0.0
0.1
0.2
0.3
0.4
0.5
0.6
bits per pixel (bpp)
bits per pixel (bpp)
(a)
(b)
Figure 7: Comparison of compression efficiency on WildTrack dataset with seven views.
6.2 CODING COMPLEXITY
Figure reports the coding latency of different codecs on Intel Xeon Gold 6230R processor with a single CPU core. For the proposed methods, we also evaluate the inference latency on workstation with NVIDIA RTX 3090 GPU. On the CPU platform, the proposed methods achieve tremendous encoding speedup improvement against VVC, which benefits from parallel processing all images in the DSC architecture. Because of the auto-regressive model and computation resources constraint, the decoder has large latency. The proposed framework targets for applications related distributed camera systems, such video surveillance and multi-view image acquisition. These applications require a low-power encoder, while the receiver has powerful computation resources to support decoding procedure. As depicted Figure 8(b) the proposed-fast variant on the GPU platform consumes less decoding time and outperforms HEVC with 14.14% bitrate saving measured by PSNR. When compared with VVC, the fast variant with only 0.86% increase bits reduces about 50% decoding time. The results demonstrate that the decoding latency of proposed methods with GPU support can meet the basic needs.
InStereo2K Dataset
InStereo2K Dataset
103
Encoding time (s)
Tropceed Pebeeed-aaatthh faat
307
Tropcees (Pal)
Decoding time
107
109
10
0.1
0.2
0.3
0.4
0.5
0.1
0.2
0.1
0.4
bits per pixel (bpp)
bits per pixel (bpp)
(a)
(b)
Figure 8: Encoding and decoding time of proposed methods and traditional codees on InStereo2K dataset.
6.3 VISUALIZATIONS
In Figure present several examples to vividly compare the quantitative results among Cheng2020, VVC, NDIC and the proposed method. It is observed that our proposed method fectively restores the image details and maintains higher reconstruction quality while consuming the
14

---

Published as a conference paper at ICLR 2023
Average bpp 369
Averge bppm0.338
Average 9pp-0.530 538
Average bpp 0.293
PSNRR-36
PSiNToeadd
PSNR
PSNR
Average bpp-0.0948
Average bog 0.074
Average bpp 0.134
Average bpp- 0.089
PSh
PSND=l
Average bpp-
Average bop=
Aperage bpp=
Average byg-
PENR HAAU
FSNR:
PENR
(a) Ground truth
(b) Cheng2020
(c) VVC
(d) NDIC
(e) Proposed method
Figure 9: subjective comparisons on the InStereo2K, Cityscapes and WildTrack datasets, where the best results are outlined in red color.
lower bits on the InStereo2K and WildTrack datasets. Similar the results Figure 4(b) achieves the best coding gain the Cityscapes dataset.
VVC
6.4 FOUNDATIONS OF SYMMETRIC DISTRIBUTED SOURCE CODING
The formal statements of Slepian- Wolf theorem (Slepian
Wolf
1973 Wolf
1973) and
Berger-Tunn proposition (Berger 1978 Tung 1978 Servetto 2006) are follows.
Theorem (Slepian-Wi) Wolf) Let X1 and X₂ be two statistically dependent i..dd discrete sources.
The achievable rate region of independently encoding X1 and X2 with joint decoding under lossless compression is specified by:
Rí H(X1]X2), Rr > H(x2|X1), Ri Rs H(X1, X2):
where R₁ and R2 are the rates for representing X1 and X2, respectively.
Proposition 1 BBerger-Tung Bound) Let U₁ and U2 be auxiliary variables such that there ex- ist decoding functions X1 fi and X₂ U2). Given the distortion con- straints E E X,,, < Dj. j = 1,2, the rates (R1,R2) follows the rate region R₁ 2 I(X,, X2; U2), Ra 2 I(X1, U2U RA + Rs 2 I(X1, U2), for some joint dis-
tribution p(z1, 32, U1, W2).
Inner Bound: when p(11, 32, U1, Uz) satisfies Markov chain Un X1 X2 U2 all rates (R1, R2)) are achievable.
Outer Bound: when p( F2, U1 u2) satisfies Markov chain U2 X1 X2 and X1
X₂ - U2, those rate points outside the union composed of the set of rates defined for each such p(21, 32, u1, W2) are not available.
15

---

Published as a conference paper at ICLR 2023
The Slepian- Wollf theorem and Berger-Tung bound proposition investigate the lossless and lossy compression of two correlated sources with separate encoders and a joint decoder, respectively. Although until now the compression limit of symmetric coding in the lossy case is still open, these theoretical results indicate that it is possible to compress two statistically dependent signals a distributed way while approaching the compression performance of joint encoding and decoding.
6.5 EXPERIMENTAL DETAILS
Dataset. We take two public stereo image datasets, InStereo2K (Bao et al., 2020) and Cityscapes (Cordts et al. 2016), and a multi-camera dataset, WildTrack (Chavdarova et al. 2018), for evalua- tion. The InStereo2K dataset involves 2060 image pairs for close views and indoor scenes, where 2010 and 50 pairs are selected as the training and testing data, respectively. The Cityscapes dataset is comprised of 5000 image pairs for far views and outdoor scenes, which is categorized into 2975 training, 500 validation and 1525 testing pairs. For the WildTrack dataset, we use FFMPEG to extract the images from seven HD 1080 videos at one frame per second. We choose the first 2000 images and the remaining 51 images in each view for training and testing. During evaluation, we minimally crop each image on the InStereo2K dataset that both height and width multiples of 64. As for the Cityscapes dataset, we follow the same cropping operations in WWddinger et al. (2022) to remove rectification artefacts and ego-vehicle, where 64, 256 and 128 pixels from the top, bottom, and sides in each image are cut off.
Traditional baseline codecs. We use the evaluation script from CompressAl to obtain the results of conventional codecs. Specifically, instead of using the default x265 encoder in BPG, we adopt the slower but efficient JCTVC encoder option to achieve the higher compression performance. For HEVC and MV-HEVC, the results on the stereo image datasets come from Wödlinger et al. (2022). We use 1M-16.25 and HTTM-16.3 softwares softwaresss to evaluate the coding efficiency HEVC and MV- HEVC on the WildTrack dataset, respectively. In addition, run VTM-17.0 to test VVC-intra and VVC.
Learning-based benchmarks. In DNN-based stereo image codecs, we retest HESIC and HESIC+ including the post-processing network by using their open source codes o because they previously reported the wrong results in their paper. The results of DSIC, BCSIC and SASIC are quoted from their corresponding papers. BCSIC did not report the rate-distortion points on the Cityscapes dataset. For distributed models, NDIC is composed of different models, where is a single image codec used in Ballé et al. (2018). another consists of separate encoder and joint decoder with side information proposed in Mital al. (2022b).
Architecture details. Details about the network layers in our framework with auto-regressive en- tropy model are outlined in Figure and 3 For the multi-head attention of the JCT module, we set the number of head as 2. The channel dimensions of the key and value are taken as one eighth and a quarter of input channels (i.e., 48 and 24), respectively. In order achieve faster coding speed, the proposed fast variant replaces the serial auto-regressive entropy model with the parallelization- friendly checkerboard entropy model in He et al.) (202T), which has the same network architecture as Figure 2 except that the masked convolution layer uses a checkerboard mask.
Ablation study details. Based on the proposed method, we insert two JCT modules after the second and fourth convolution layers at the encoder to implement the Joint Enc-Dec, thereby allowing both the encoder and the decoder to access the inter-view context. For the Sep Enc- the JCT modules at the decoder are removed, making it equivalent to single image compression. These models are trained hased on the InStereo2K dataset by using the same training scheme as LDMIC (See Implementation Details in Section 4. encoder and entropy model on the Sep Enc-Dec, and only train the joint decoder on the InStereo2K dataset, which follows the same training procedure as in our proposed method.
Tp. For the W/O Join! Training case, we fix the pre-trained
hhtpss//vcgitthii.fraa wtt.:h...uuubbee.e.e//////tt----
16


Reviews:
[{"review_id": "jdgMysJTSo", "reviewer": "Reviewer_WX5p", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces a learning-based distributed multi-view image coding (LDMIC) method. LDMIC consists of independent encoders and a decoder equipped with a cross-attention mechanism based joint context transfer module. The proposed method does not need synchronization between cameras and is insensitive to the epipolar geometry relations between images. Experimental results show that LDMIC outperforms existing multi-view image coding methods in saving bitrate.\n", "summary_of_the_review": "This paper introduces a simple yet effective method for distributed multi-view image coding, and experiments demonstrate the effectiveness of the method. I would like to give a weak accept rating. However, one of my concern is that the problem tackled by this paper (multi-view image coding) seems to be very specific and only attract a limited amount of audiences in ICLR. ", "strength_and_weaknesses": "Strength\n- This paper introduces a distributed multi-view image coding framework that decouples the inter-view operations at the encoder, which is a good property in a distributed camera system.\n- This paper proposes a joint context transfer module at the decoder to capture inter-view correlations for image reconstruction.\n- Extensive experimental results show that the proposed method outperforms existing single and multi-view image codecs in PSNR.\n\nWeaknesses\n- The authors claim that the proposed joint context transfer module \"can serve as a plug-and-play component to enhance the performance of learning-based MIC methods\" in the last paragraph of Page 2. However, no supporting experiments are shown\n- It appears to me that the technical part of the proposed method is straightforward and not very novel.\n- It is unclear what is the fast variant of the proposed method. Is it the one using the checkerboard model? Please clarify.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "9ldIW8LbjJm", "reviewer": "Reviewer_XGBn", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposed a learning-based approach for multi-view image coding. Specifically, the proposed method decouples the inter-view operations during the encoding and thus is suitable for the distributed system. Thorough evaluations demonstrate the effectiveness of the proposed method in terms of PSNR and SSIM.", "summary_of_the_review": "This paper achieves the best performance on the task of multiview image coding. However, I do not see many technical novelties in the method or architecture design. The most exciting part is that one can use the proposed method for the distributed system. I lean toward accepting this paper but rejecting it is also acceptable.", "strength_and_weaknesses": "- Strengths:\n  - This paper is well-written and easy to follow.\n  - The proposed method performs favorably against existing image compression methods.\n  - The proposed protocol for the distributed system is interesting and useful.\n- Weaknesses:\n  - Since this task is mainly about compressing multiview images, there should be some priors that can be leveraged. Such as epipolar constraints. However, I do not see any special design of the proposed method in addition to handling multi-frame. Is there any reason for not using these priors to encode multiview images better?\n  - I do not see many technical novelties in this paper. The proposed architecture is similar to VQVAE and VQGAN. The authors do not specifically point out how you integrate the existing architecture into this task and why does it help to get better compression results?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "72fPIGEaTc", "reviewer": "Reviewer_8AGJ", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors proposed a learning-based multi-view coding framework comprising of  individual encoding and joint decoding of a set of multi-view images. Under their framework, inter-view coherency can be used for improving the coding performance while eliminating the need of communication among the distributed encoders for individual views. The proposed method was designed to be symmetric: each view is treated equally, and independent of the number of viewpoints. To this end, they incorporated joint context transfer modules into the decoder, where the features from a specific viewpoints are enhanced by those from the other viewpoints via the multi-head cross attention module. The experimental results include extensive comparisons with previous works in terms of the coding performance and computational efficiency, ablation studies for verifying their distributed framework and joint decoding design, and subjective evaluations.", "summary_of_the_review": "The paper is well written, the authors' proposal has sufficient novelty, and experimental results are extensive and informative for the community. However, I have several concerns for the correctness (Fig. 4 (c) and (f)) and clarity. I would like to recommend acceptance for this work, but not strongly so. ", "strength_and_weaknesses": "Strengths\n\n- The paper is well written and easy to follow. The story (background) behind this work is well composed in a succinct manner. The proposed method is clearly motivated and well explained to the details. \n\n- Although there are some previous works on distributed coding of multi-view images, this is the first work to develop a view-symmetric framework with deep learning and achieve the coding performance comparable to the SOTA joint encoding-decoding schemes.\n\n- The experimental evaluation is extensive with convincing results. Especially, the proposed method clearly outperforms NDIC (the method closest to the authors') in terms of the coding efficiency. Moreover, the presented results include many insights: the effectiveness of inter-view cross attention compared to the conventional disparity-based prediction, the advantage of the view-symmetric design over the asymmetric one, and the effectiveness of the joint encoder-decoder training for making the latent representation more efficient.\n\nWeaknesses\n\n- I have a concern for Fig. 4 (c) and (f). The authors state that these graphs are drawn for the selected cameras (C1 and C4). However, most of the methods presented in these graphs take joint decoding or joint coding-decoding scheme, where the information from all the cameras are jointly used. Therefore, it is theoretically impossible/meaningless to define the bitrate only for the selected cameras. \n\n- As the authors admitted, a simple average pooling for merging multi-view information is not sufficient enough; as shown in Table 3, increasing the number of views did not much contribute to the reconstruction quality.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "3qnhEmZKOY", "reviewer": "Reviewer_11ke", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposed a novel deep learning based framework to effectively encode the distributed multi-view images. Unlike previous multi-view image coding (MIC) architectures that utilized a shared encoder to explore the inter-view redundancy, the proposed method moved this part to decoder and successfully reduced the computational costs in MIC. The experiments demonstrate the robustness and efficiency of proposed method.", "summary_of_the_review": "In summary, this paper proposed a novel coding algorithm that can decouple the encoding of images acquired by distributed cameras. The method seems to effective and the results are good. The paper is well written and the comparison experiments are comprehensive. ", "strength_and_weaknesses": "Strength:\n+ The paper is well written and easy to follow\n+ The results are good compared with SOTA\n+ The experiments are comprehensive and well evaluate the performance on different datasets\n\nWeakness:\n- It will be great if there's a short discuss of failure cases. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster