On the Out-of-distribution Generalization of Probabilistic Image Modelling
Authors: Mingtian Zhang
Andi Zhang
Steven McDonagh
Venue: NeurIPS 2021 Poster (2021)
Keywords: OOD detection
OOD generalizationm
Lossless compression
Generative model

Abstract: Out-of-distribution (OOD) detection and lossless compression constitute two problems that can be solved by the training of probabilistic models on a first dataset with subsequent likelihood evaluation on a second dataset, where data distributions differ. By defining the generalization of probabilistic models in terms of likelihood we show that, in the case of image models, the OOD generalization ability is dominated by local features. This motivates our proposal of a Local Autoregressive model that exclusively models local image features towards improving OOD performance. We apply the proposed model to OOD detection tasks and achieve state-of-the-art unsupervised OOD detection performance without the introduction of additional data. Additionally, we employ our model to build a new lossless image compressor: NeLLoC (Neural Local Lossless Compressor) and report state-of-the-art compression rates and model size.

On the Out-of-distributio Generalization of Probabilistic Image Modelling
Mingtian Zhang 1,2+
Andi Zhang 2,3+
Steven McDonagh 2
1 AI Center, University College London, Huawei Noah's 's Ark Lab, 3 Department of Computer Science and Technology, University of Cambridge steven stteen. www.eemccoona@hhuuee...com
mingtian. zhang. 1ww0177.lc..m
az3818cam. ac. uk
Abstract
Out-of-distributien (OOD) detection and lossless compression constitute two prob- lems that can be solved by the training of probabilistic models on a first dataset second dataset, where data distribu-
with subsequent likelihood evaluation tions differ. By defining the generalization of probabilistic models terms likelihood we show that, in the case of image models, the OOD generalization ability dominated by local features. This motivates our proposal Autoregressive model that exclusively models local image features towards im- proving OOD performance. We apply the proposed model to OOD detection tasks and achieve state-of-the-arr unsupervised OOD detection performance without the introduction of additional data. Additionally, we employ our model to build a new lossless image compressor: NeLLoC (Neural Local Lossless Compressor) and report state-of-the-art compression rates and model size.
Local
Introduction
Probabilistic modeling has achieved great success in the modeling of images. Likelihood based models, e.g. Variational Auto-Encodert (VAE) 22 Flow (21]. Pixel CNN 49 are shown to successfully generate high quality images, in addition to estimating underlying densities.
The goal of probabilistic modelling is to model pa approximate the unknown data distribution ~ Pa- A common method to learn parameters to minimize some divergence between pa and po. for example, a popular choice is the KI. divergence
Pa using the training data (.1,... " .NN
KT.(pallpg ((ppallpo) H(p2) / log Tog (o)))ap(())dd
(1)
where the entropy of the data distribution is a constant. Since we only have access to finite pa data samples 21...., NN, the second term is typically approximated using Monte Carlo estimation
(LLpallpa) ~0
log pe po(x) const..
(2)
1=1
Minimizing the KI. divergence is therefore equivalent Maximum Likelihood Estimation. A typical evaluation criterion for the learned models is the test likelihood 1
M ΣM log (o with test set (z1...., .Mr} } ~ Pd. We refer to this evaluation as in-distribution (ID) generalization, since both training and test data are sampled from the same distribution Pa. However, in this work, we are interested in out-of-diittrbuutinn (OOD) generalization such that the test data are drawn from po. where p. f Pa- To motivate our study this topic, we firstly introduce two applications: lossless compression and OOD detection, that both can make use of the OOD generalization property.
"Co-first author, the work was done during an internship in Huawei Noah's Ark Lab.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

1.1 Model-based Lossless Compression
Lossless compression has a strong connection to probabilistic models 33 Let { [MM be test data to compress, where Im is sampled from some underlying distribution with PMF pa- If Pa is known, we can design a compression scheme to compress each data Z'm. to bit string with length approximately - logg pu(xm As M + oo, the averaged compression length will approach the entropy of the distribution Pa. that is; M ΣM .oo..--og,,, logg Pa(am) - H(ps) where H(-) denotes entropy. In this case, the compression length is optimal under Shannon's 's source coding theorem [5]] i.e. we cannot find another compression scheme with compression length less than Hppa). In practice, however, the true data distribution Pa is unknown and we must use model Po RJ Pa to build the lossless compressor. Recent work successfully apply deep generative models such as VAFs [7 48 23 Flow [19 3 to conduct lossless compression. We note that the underlying models are designed to focus on test data that follows the same distribution as the training data, resulting in test compression rates that depend the ID generalization ability of the model.
However, in practical compression applications, the distribution of the incoming test data unknown, and is usually different from the training distribution pa: Xtest = ( Jas where x' ~ Po + pa- To achieve good compression performance, a lossless compressor model should still be able assign high likelihood for these OOD data. This practical consideration motivates encouragement of the OOD generalization ability of the model.
Empirical results 48 19 have shown that we can still use model pe (trained to approximate pa) in order to compress test data Nestt with reasonable compression rates. However, the phenomenon that these models can generalize to OOD data lacks intuition and key components, affecting this generalization ability, remain underexplored. Consideration of recent advances in likelihood-baat basd OOD detection next allows us to further investigate these questions and lead to our proposal of a new model that can encourage OOD generalization.
1.2 Likelihood-baas OOD Detection
Given set of unlabeled data, sampled from pa. and test data x' then the goal of OOD detection is to distinguish whether or not x' originates from pa- A natural approach [5] involves fitting a model Pa to approximate Pa and treat sample x' in-distribution if its (log) likelihood is larger than threshold; log (x²) > e. Therefore, a good OOD detector model should assign low likelihood to the OOD data. In contrast to lossless compression, this motivates discouragement of the 0OD generalization ability of the model.
Surprisingly. recent work 36) report results showing that popular deep generative models, including e.g. VAE [22), Flow [21] and PixeICNN AT. can assign OOD data higher density values than in-distribution samples, where such OOD data may contain differing semantics, c.f. the samples used for maximum likelihood training. We demonstrate this phenomenon using PixeICNN models, trained on Fashion MNIST (CIFAR10) and tested using MNIST (SVHN). Figure I provides histograms of model evaluation using negative bats-per-dimension (BPD), that is; the log, likelihood normalized by data sample dimension (larger negative BPD corresponds larger likelihood). We corroborate previous work and observe that tested models assign higher likelihood to the OOD data, in both cases. This counter intuitive phenomenon suggests that rikelihood-basee approaches may not make for good OOD image detection criterion, yet encouragingly also illustrates that probabilistic model, trained using one dataset, may be employed compress data originating from a different distribution with a potentially higher compression rate. This intuition builds a connection between OOD detection and lossless compression. Inspired by this link, we next investigate the underlying latent causes of image model generalizability, towards improving both lossless compression and OOD detection.
2 OOD Generalizations of Probabilistic Image Models
Previous work studies the potential causes of the surprising OOD detection phenomenon: OOD data may have higher model likelihood than ID data, For example, (37 used a typical set to reason about the source of the effect, while the work of [40] argues that likelihoods are significantly affected by
FFor a compression method like Arithmetic Coding 50 the message length is always within two bits of logg pa()m) (33), also Section 4.1

---

FMNIST
MNIST
Figure 1: Left: log likelihood of FashionMNIST, MNIST test data using a full PixeICNN model, trained on FashionMNIST training set. Right: log likelihood of CIFARIO, SVHN test data using a PixelCNN model, trained on CIFARIO training set. The z-axis indicates the value of the log- likelihood (negative BPD), the y-axis provides data sample counts.
image background statistics or by the size and smoothness of the background [26]. In this work, we alternatively consider a recent hypothesis proposed by 43 (also implicitly discussed in [24] for a flow-based model): low-level local features, learned by (CNN-based) probabilistic models, are common to all images and dominate the likelihood. From the perspective of OOD generalization, this allows formulation of the following related conjectures:
Models can generalize to OOD images as local features are shared between image distributions.
Models can generalize well OOD images since local features dominate the likelihood.
In the work of 43 the authors investigated their original hypothesis by studying the differences between individual pixel values and neighbourhood mean values and additionally considered the correlation between models trained small image patches and trained, alternatively, To further investigate this hypothesis, we rather propose to directly model the in in-distribution dataset, using only local feature information. If the hypothesis true, then the proposed local model alone should generalize well to OOD images. By contrasting such an approach with a standard full model, that considers both local and non-local features, we are also able study the contribution that local features make to the full model likelihood. We first discuss how to build a local model for the image distribution and then use the proposed model study generalization on OOD datasets.
full images.
2.1 Local Model Design
Autoregressive models have proven popular for modeling image datasets and common instantiations include PixeICNN [49 4I], PixeIRNN [49] and Transformer based models (7]. Assuming data
dimension D, the Autoregressive model py (z) can be written as
D
py(x) = p(x1) II
Id-1).
(3)
d=2
informally we refer this type of model between each dimension (pixel). Similarly, we can define a local autoregressive model pr( where pixel Wej, image row column j, depends on previous pixels with fixed horizon h;
"full model" since it capture all dependencies
pr(x) II p(a ( Tits -i:--)))
(4)
with zero-padding used in cases where i or are smaller than h. Figure 2b illustrates the resulting local autoregressive model dependency relationships. We implement this model using masked CNN (4) with kernel size k=2x/++ h++11 in our first network layer, to mask out future pixel dependency. A full PixeICNN model would then proceed to stack multiple masked CNN layers, where increasing kernel depth affords receptive field increases. In contrast, we employ masked CNN with 1x 1 convolutions in subsequent layers. Such 1 Xx] 1 convolutions can model the correlation between channels, as 21], and additionally prevent our local model obtaining information from pixels outwith the local feature region, defined by h. Pixel dependencies are therefore defined solely using the kernel size of the first masked CNN layer, allowing for easy control over model local feature size. We note that the proposed local autoregressive model can also implemented using alternative backbones e.g. Pixel Unlike full autoregressive models 49 41, which can be used to generate semantically coherent samples, we find the samples from the local model are locally consistent yet have clear semantic meaning.
RNN 49 Transformers 7 We plot local model samples in Figure 4
3

---

(a) Full Autoregressive Model
(b) Local Autoregressive Model
Figure 2: (a) full autoregressive model pixel dependencies; the distribution current pixel (blue) depends all previous pixels (red); (b) local autoregressive model dependencies, with h = 2. The distribution of Tiy (blue) depends on only the pixels in a local region (red).
2.2 Local Model Generalization
To investigate the generalization ability of our local autoregressive model, we fit the model to Fashion MNIST (grayscale) and CIFARIO (color) training datasets and test using in-distribution (ID) images (respective dataset images) and additional out-of-distribution (OOD) datasets: MNIST, KMNIST (grayscale) and SVHN, CelebA" Both models use the discretized mixture of logistic distributions 41 with 10 mixtures for the predictive distribution and ResNet architecture 44
We use a local horizon length h=3 (kernel size k=7) for both grayscale and color image data. We compare our local autoregressive model standard full autoregressive model (i.e. a standard PixeICNN), with additional network architecture and training details found in Appendix A Tables
Table 1: Test BPD (Trained on Fashion MNIST)
Table 2: Test BPD (Trained on CIFAR10)
Test Dataset
Full
Local
Test Dataset
Full
Local
Fashion MNIST (ID)
2.78
2.89
CIFARIO (ID)
3.12
3.25
MNIST (OOD)
1.50
1.49
SVHN (OOD)
2.13
2.13
KMNIST (OOD)
2.48
2.44
CelebA (OOD)
3.33
3.35
report comparisons in terms of BPD (where lower values entail higher likelihood) for Fasion MNIST and CIFAR 10, respectively. We observe that for in in-distribution (ID) data, the full model has better generalization ability c.f. the local model (0. and 0.13 BPD, respectively). This is unsurprising as training and test data originate from the same distribution; both local and non-local features, as learned by the full model, help generalization. For OOD data, we observe that the local model has generalization ability similar to the full model, exhibiting very small empirical gaps (only z 0.02 BPD on average), showing that the local model alone can generalize well to OOD distributions. We thus verify the hypothesis considered at the start of Section 2
For simple datasets containing gray-scale images, the PixeICNN model flexible enough to capture both local and global features. We notice that, in Table I our local model exhibits even better OOD generalization than the full model. This drives us further study the role non-local features for generalization. When the local horizon size increases, the model will be able to learn features with greater non-locality. We thus vary the local horizon size to study generalization ability under this property, see Table [ We find the model has poor generalization performance when local features are too small and increasing the horizon size helps ID generalization but decreases the OOD generalization. A consistent phenomenon is observed when considering color images, see Appendix B. We can thus conclude: non-local features are not shared between images distributions, overfitting to non-local features will hurt generalization.
These hypotheses indicate two opposing Table 3: Generalization of local model with different hori- strategies for the considered tasks:
sizes. The model is trained on MNSHI dataset.
distributions distin- guished by dataset-unique features, thus building non-local models, able to dis- count common local features, improves the detector distinguishability power.
0OD detection:
Method
h=1
h=2
h=3
h=4
h=5
(ID) Fashion
3.17
2.93
2.89
2.88
2.88
(OOD) MNIST
1.54
1.48
1.49
1,50
1.51
(OOD) KMNIST
2.54
2.43
2.44
2.46
2.47
We down- down-sample the original CelebA to 32x32x3, Appendix
A.2 for details.

---

Lossless compression: OOD generalization is possible due to the sharing of local features between distributions. Employing only a local model can encourage OOD generalization. by preventing the model from over-fitting to dataset-unique features, specific to the training distribution.
Sections
and will further demonstrate how contrasting modeling strategies can benefit these tasks.
3 OOD Detection with Non-Local Model
As was discussed in Section I.2 local image features are observed to be largely common across the real-world image distribution and can be treated as a domain-prior. Therefore, in order to detect whether or not an image is of-distributton, we can stipulate a non-local model able to discount local features of the image distribution; denoted here (z). It is however not easy to build such non-local model directly, since the concept of "non-local" lacks a mathematically rigorous definition. However, we propose that non-local model can be considered to be the complement of local model, from a respective full model. In the following section, we therefore propose to use a product of experts to indirectly define (x), and demonstrate how this may be used for OOD detection.
3.1 Product of Experts and Non-Local Model
As demonstrated Section 2.2 the full model py (z))) and the local model pi (ar) can be easily built for the image distribution, e.g. full autoregressive model and a local autoregressive model. We further assume the full model allows the following decomposition:
Pf (a) ((i)))
(5) where Z f (x (x) is the normalizing constant. This formulation can also be thought of as a product of experts (PoE) model 17 with two experts; pt (local expert) and pmd (non-local expert). An interesting property of the PoE model is that if a data sample x' has high full model probability (a'),, should possess high probability mass in each of the expert components Fl Therefore, the PoE model assumption is consistent with our image modelling intuition: valid image requires both valid local features (e.g. stroke, texture, local consistency) and valid non-local features (semantics).
2
By our model assumption, the density function of the non-local model can be formally defined and is proportional to the likelihood ratio:
Pr(x) (n)( p((x))
(x)) OX
(6)
where (x) denotes the unnormalized density. For the OOD detection task, we require only Paa (x) in order provide score classifying whether not test data is OOD and therefore do not require to estimate the normalization constant Z. We also note that as we increase the local horizon length for pt. the local model will converge to full model pt + Pf. and (x) 1 becomes a constant and inadequate for OOD detection. This further suggests the importance of using local model. Figure 3 shows histograms of (-) for both ID and OOD test datasets. We observe that the majority of ID test data obtains higher likelihood than OOD data, illustrating the effectiveness of non-local models.
3.2 Connections to Related Methods
We highlight that the score (x) that use to conduct OOD detection allows a principled likelihood interpretation: the unnormalized likelihood of a non-local model. We believe this to be the first time that the likelihood of non-local model is considered in the literature. However, other likelihood ratio variants have been previously explored for OOD detection. We briefly discuss related work and highlight where our method differs from relevant literature.
In 400, assumed that each data sample x can be factorized as = (25, where I'b is a background component, characterized by population level background statistics: Pb. Further,
"This PoE property differs from a Mixture of Experts (MoE) model that linearly combines experts. If data o' has high probability in the local model (e.g- pr(x') 0.9) but low probability in the non-local model (e.g- Pez (I²) 0.1), the probability in the full PoE model py px(')) x 0.9 0.1 is also small. On the contrary, if 0.1 results a high full MoË model pr(x') value i.e. pr(x') 0.5 0.9 0.5 0.1 0.5. We refer to 17 for additional PoE model details.
assume py is MoE: py fp + Put, then pi (x') 0.9 and p=t(=")
5

---

FMNIST
CIFAR
MNIST
-0.2 0
0.2
0.4 0.6
0.2
Figure 3: Unnormalized log likelihood of the non-local model (ni(2) (z) Pr(a) for both ID test datasets (FashionMNIST, CIFARIO) and OOD datasets (MNIST, SVHN). ID test datasets obtain signifi- cantly higher likelihoods, on average, in each case.
Figure 4: Samples from the lo- cal autoregressive model, see Appendix B.2 for details.
then constitutes semantic component, characterized by patterns belonging specifically to the in- distribution data: ps- A full model, fitted on the original data, (e.g. Flow) can then be factorized as ()(x)) Py (Tz, x Po(#))p,, (x..) and the semantic model can correspondingly be defined as
ratio: (x) = (r(#)) where py (z:) is a full model. In order to estimate po(zz), the authors of [40]
(2))
design perturbation scheme and construct samples from (x(2) by adding random perturbation to the input data. A further full generative model is then fitted to the samples towards estimating Pb- In our method, both (z)) and P.t (z) constitute distributions of the same sample space (that of the original image 2:) whereas ps and pt, in [40] form distributions in different sample spaces (that of x, and 2b, respectively). Additionally, in comparison with our local and non-local experts factorization of the model distribution, their decomposition of an image into 'background' and 'semantic' parts may not be suitable for all image content and the construction of samples from pb(2%) (adding random perturbation) lacks principled explanation. In [4 43), the score for OOD detection is defined as s(x) pr(z)
Pe(a) where py is a full model and Pe is complexity measure containing image domain prior. In practice, Pe is estimated using traditional compressor (e.g. PNG or FLIF), model trained on an additional, larger dataset in an attempt to capture general domain information [43]. In comparison, our method does not require the introduction of new datasets and our explicit local feature model, Pt. can be considered more transparent than PNG of FLIF, Additionally, our likelihood ratio can be explained as the (unnormalized) likelihood of the non-local model for the in-distribution dataset, whereas the score described by 45 44) does not offer likelihood interpretation. In Section 4 we discuss how the proposed local model may be utilized to build a lossless compressor, further highlighting the connection between our OOD detection framework and traditional lossless compressors (e.g. PNG or FLIF). T we report experimental results showing that a lossless compressor based on our model significantly improves compression rates c.f. PNG and FLIF, further suggesting the benefits of the introduced OOD detection method.
3.3 Experiments
We conduct OOD detection experiments using four different dataset-pairs that are considered chal- lenging 36): Fashion MNIST (ID) vs. MNSIT (OOD); Fashion MNIST (ID) vs. OMNIGLOT (OOD); CI (ID) vs. SVHN (OOD); CIFARIO (ID) vs. CelebA (OOD). We actively select not to include dataset pairs such as CIFAR 10 CIFARI00 or CIFARIO ImageNet since these contain duplicate classes and cannot be treated as strictly disjoint (or OOD) datasets [44]. Additional experimental details are provided in Appendix A In Table 4 we report the 'area under the receiver operating characteristic curve' (AUROC), a common measure for the OOD detection task [15]. We compare against methods, some of which require additional label information"] or datasets. Our method achieves state-of oh-theeart performance in most cases without requiring additional information. We observed that the model-ensemble methods, WAIC [8] and MSMA [34] can achieve higher AUROC in the experiments involving color images yet are significantly outperformed by our approach in the case of grayscale data. We thus evidence that our simple method is consistently more reliable than alternative approaches and that our score function allows a principled likelihood interpretation.
IIn principle methods that require labels correspond to classification, task deeeddeendent OOD detection, which may be considered fundamentally different from task independent OOD detection (with access to only image space information), II for details. We compare against both classes of method, for completeness.

---

Table OOD detection comparisons (AUROC). Higher values indicate better performance, results are rounded to three decimal places. Results are reported in each case directly using the original references except in the cases of ODIN 40 29 and VIB [ Results for Typicality test are from [4 corresponding to batches of two samples of the same type. (a) The Mahalanobis method requires knowledge of the validation data (OOD distribution). (b) A full PixeICNN (see Appendix A
trained on the ID dataset and its likelihood evaluations then used to calculate AUROC.
ID dataset:
FaShionMNIST
CIFARIO
OOD dataset:
MNIST Omniglot
SVHN
CelebA
Using Labels
ODIN 31
0,697
0.966
VIB 2
0,941
0.943
0.528
0.735
Mahalanobis
16
0,986
0,991
Gram-Matrix
42
0,995
Using Additional Datasets
Outlier Exposure 16
0.758
0.615
Glow diff to Tiny Glow
45
0,939
PCNN diff Tiny- PCNN
43
0.944
Not Using Additional Information
WAIC (model ensemble)
8
0.766
0.796
1.000
Glow diff to PNG [3
0.754
PixcICNN diff to PNG
43
0.823
Likelihood Ratio in 40
0.997
0,912
MSMA KD Tree 34
0.693
0,991
S using Glow and FLIP
45
0.998
1.000
0,950
0.736
S using PCNN and FLIF
4
0.967
1.000
0.929
0.535
Full PixeICNN likelihood"
0.074
0.361
0.113
0,602
Our method
1.000
1.000
0.969
0.949
Lossless Compression with Local Model
are designed under the assumption that data to be compressed originates from the same distribution (source) as model training data. However, in practical scenarios, test images may come from a diverse set of categories or domains and training images may be comparatively limited [33] Obtaining single method capable of offering strong compresson performance on data from different sources remains an open problem and related study involves consideration of "universal"" compression methods (33]. Based our previous intuitions relating to generalization ability; to build such a "universal" compressor in the image domain, believe promising route involves leveraging models that only depend on common local features, shared between differing image distributions. We thus propose a new "universal" lossless image compressor: NeLLoC (Neural Local Lossless Compressor), built upon the proposed local autoregressive model and the concept of Arithmetic Coding [50]- In comparison with alternative recent deep generative model based compressors, we find that NeLLoC has competitive compression rates on diverse set of data, yet requires significantly smaller model sizes which in turn reduces storage space and computation costs. We further note that due to our design choices, and in contrast to alternatives, NeLLoC can compress images of arbitrary size.
Recent deep generative model based compressors
23 18
In the remaining parts of this section, firstly discuss NeLLoC implementation and then provide further details on the most important resulting properties the method.
4.1 Implementation of NeLLoC
Our Lo implementation uses the same network backbone as that of our OOD detection experi- ment (Section 3) a Masked CNN with kernel size k x h + (h is the horizon size) the first layer and followed by several residual blocks with 1 convolution, see Appendix A for the network
7

---

architecture and training details. To realize the predictive distribution for each pixel, we propose to use discretized Logistic-Uniform Uniform mixture distribution, which we now introduce.
Discretized Logistic-Uniform Mixture Distribution The discretized logistic mixture distribution, proposed by [4]] has shown promising results for the task of modeling color images. In order to ensure numerical stability, the original implementation can provide only an (accurate) approximation and therefore cannot be used for our task of lossless compression, which requires exact numerical evaluation. We therefore propose to use the discretized Logistic-nniform Mixture distribution, which mixes the original discretized logistic mixture distribution with a discrete uniform distribution:
T x~ (1-
Σ ग. Logistsii((u; s:)
+ au((,, (U.. 255),
where U(O,.... 255) is the discrete uniform distribution over the support {0.... 255}. The proposed mixture distribution can explicitly avoid numerical issues and its PMF and CDF can be easily evaluated without requiring approximation. We can then this CDF evaluation in relation to Arithmetic Coding. In practice; we set a = 10-4 to balance numerical stability and model flexibility. We use K- 10 (mixture components) for all models in the compression task.
Coding Arithmetic coding (AC) (50) is a form of entropy encoding which utilizes a (discrete) probabilistic model p(-) to map datapoint x to interval [o, 1]. One fraction, lying in the interval, can then be used to represent the data uniquely. If we convert the fraction into a large message stream, the length of the message is always within two bits of - logg p(x) [33). Our vanilla NeLLoC implementation makes use of the decimal version of AC, In principle, NeLLoC can also be combined with an Asymmetric Numeral System (ANS) [.Л. which gives faster coding time yet sacrifices compression rate. The work of [13) further proposed interleaved-ANS (iANS), which enables coding batch of data simultaneously. In Appendix O
Arithmetic
we report comparison of NeLLoC-AC, NeLLoC-ANS and NeLLoC-iANS in terms of compression rate and run time f
4.2 Properties of NeLLoC
Universal Image Compressor As discussed previously, the motivation for designing NeLLoC is to realize an image compressor that is applicable (generalizable) to images originating from differing distributions. Towards this, NeLLoC conducts compression depending on local features which are shown to constitute a domain prior for all images. In addition to universality properties, we next discuss other important aspects towards making NeLLoC practical when considering real applications.
Arbitrary Image Size Common generative models, e.g. VAE or Flow, can only model image distributions with fixed dimension. Therefore, lossless compressors based on such models 47
19 3
18 23
can only compress images of fixed size. Recently, HiLLoC 48 explore fully convolutional networks, capable of accommodating variable size input images, yet still requires even height and width? L3C [35], based on multi-scale autoencoder, can compress large images yet also requires height and width to be even. NeLLoC is able to compress images with arbitrary size based on an alternative and simple intuition: we only model the conditional distribution, based on local neighbouring pixels. We thus do not model the distribution of the entire image and can therefore, in contrast to HiLLoC and L3C,, compress arbitrary image sizes without padding requirements.
To validate method properties,
we compare the compression performance of NeLLoC with both traditional image compressors and recently proposed generative model based compressors. We train NeLLoC with horizon length h = 3 on two (training) datasets: CIFARI0 (32 x32) and ImageNet32 (32x32) and test on the previously introduced test sets, including both ID and OOD data. We also test on ImageNet64 with size 64x64 and a full ImageNef test set (with average size 500 3774.. Table 6 provides details of the comparison. We find NeLLoC achieves better BPD in a majority of cases. Exceptions include LBB [18] having better ID generalization for CIFAR and HiLLoC (48) having better OOD generalization in the full ImageNet, when the model trained on ImageNet32.
Small Model Size In comparison with traditional codecs such as PNG or FLIF, one major limitation of current deep generative model based compressors is that they require generation and storage of
We provide practical implementations of NeLLoC with different coders and pre-trained models at attps://githu: com/ ht////ttooo www.orowwwwoeoooo.C.c..
For images with odd height or width, padding is required.
Images with height or width greater than 1000 are removed, resulting in total of 49032 test images.
8

---

models of very large size. For example. HiLLoC 48 contains 24 stochastic hidden layers, resulting in capacity and parameter size of 156 MegaBytes (MB) using 32-bit floating point model weights. This poses practical challenges relating to both storage and transmission of such models to real, often resource-limited, edge-devices. Since NeI.LoC only models the local region, a small network: three Residual blocks with x convolutions (except the first layer) is enough to achieve state-of-the-art compression performance. Accordingly the parameter size requirement is only 2.75 MB. We also investigate the compression task under NeLLoC with I and 0 residual block, which have parameter size of 1.34 and 0.49 MB respectively, and yet still observe respectable performance. We report model size comparisons in T In principle, NeLLoC can be also combined with other resource saving techniques such as binary network weights to realize the generative model components. This would further reduce model size [ which we consider a promising line of future investigation.
Computation Cost and Speed The computational complexity and speed for a neural based lossless compressor depends on two stages: (1) Inference stage: in order to compress or decompress the pixel Td, a predictive distribution of zd needs to be generated by the probabilistic models; (2) Coding stage: uses the pixel value Fa and its distribution to generate the binary code representing za (encoding) or uses the predictive distribution and the binary code to recover za (decoding). The computation cost and speed of the second stage heavily depends on the implementation (e.g. programming language) and the choice of coding algorithms, see Appendix for a discussion about how different coders trade off between speed and accuracy. We here only discuss the computational cost pertaining to the first inference stage, which is affected by two factors: computational complexity and parallelizability.
1. Computational complexity: given an image with size NxN, the full autoregressive model distribution of pixel may depends on all previews pixels p(ray (1))) and the computational cost of conditional distribution calculation scales as (N2). For latent variable models (e.g. [48] 47) or
flow models (e.g. (3) the predictive distribution of x; depends on all other pixels and therefore also scales with (N?). In direct contrast, our local autoregressive model only depends on local regions with horizon h, computation of p(zzy) (P(e-:ii-,, h.i-1.j-A:j A.jj):j+hh+-- - -1)-11) scales with only O(h)) and typically & in practice. This results in significant reduction of computational cost, enabling efficient usage of NeLLoC on CPUs. In Appendix C. we show that NeI.Lo,, with interleaved ANS coder, is able to compress or decompress the CIFARTO dataset within 0.1s per image on a CPU. This gives evidence, for the first time, that computation need not be a limitation of generative lossless compression. Further implementation improvements (e.g. C++, faster coders such as IANS (12), will make NeLLoC a strong candidate supersede traditional image compression methods.
Purallelizabildyy; second key property affecting compression speed is parallelizability. Neu- ral compressors that make use of latent variable models 47 48 can allow for the distribution of each pixel to be independent (given decompressed latents). Flow models with an isotropic base distribution will also result in appealing decompression times average. However for full autoregressive model, the predictive distribution of each pixel must be generated in sequen- tial order and thus the decompression stage cannot be directly parallelized. This results in the decompression stage scaling with image size. In contrast to a full autoregressive model, where sequential decompressing inevitable, we note that with NeLLoC each pixel only depends local region. The fact that multiple pixels exist with non-overlapping local regions allows simul- taneous decompression at each step. enabling parallelization. Alternatively, we can split an image into patches and compress each patch individually, thus parallelizing the decompression phase. 5 the compression BPD when we split full ImageNet images into patches. BPD slightly increases when we increase the num- ber of patches. Splitting an image into patches assumes each patch is independent, which weak- ens the predictive performance. This results in a trade-off between compression speed and rate: x0.1 I BPD overhead can offer 4400 speed up in compression, decompression (assuming in- finite compute), potentially proving extremely valuable for practical applications.
Table 5: Parallelization using patches on full Im- ageNet. The model uses h 3, r trained on CIFAR1O. We use e.g. '400x" to denote an image is split into 400 patches. The reported BPD has standard deviation ~0.02 across multiple random seeds.
and is
Method
1x
16x
25x
100x
400x
BPD
3.25
3.26
3.27
3.29
3.34

---

Table Lossless Compression Comparisons. We compare against traditional images compression and neural network based models. For neural models, we report results where models are trained on CIFAR or ImageNet32 and tested on other ID or OOD test datasets. We use f to represent the best ID generalization and represent the best OOD generalization. (a) We down-sample CelebA to x3,, see Appendix A (b) The BPD 3.15 reported 48 tested on 2000 random samples from the full ImageNet testset, whereas test HiLLoC the whole testset with 49032 images. The
reported BPD NeLLoC has standard deviation ~0.02 across multiple random seeds.
Method
Size(MB)
CIFAR
SVHN
CelehA
ImgNet32
ImgNe164
ImgNet
Generic
PNG 6
5.87
5.68
6.62
6.58
5.71
5.12
WebP 30
N/A
4.61
3.04
4.68
4.68
4.64
3.66
JPEG2000 39
N/A
5.56
4.10
5.70
5.60
5.10
3.74
FLIF 46
N/A
4.19
2.93
4.44
4.52
4.19
3.51
Train/test distribution
ID
ID
ID
LBB 18
3.12
3.88
3.70
IDF+
3.26
4.81
Trained on CIFAR
ID
OOD
OOD
OOD
OOD
IDF 10
223.0
3.34
4.18
3.90
Bit-Swap 23
3.78
2.55
3.82
5.37
HiLLoC 48
156.4
3.32
2.29
3.54
4.89
4.46
3.42
I.C 35
19.11
3.39
3.17
4.44
4.97
4.77
4.88
NeLLoC 0)
0.49
3.38
2.23
3.44
4.20
3,86
3.30
NeLLoC 1)
1.34
3,28
2.16
3.37
4.07
3.74
3,25
NeLLoC (r 3)
2.75
3,25
2.13
3.35"
4.02"
3.69
3,24
Trained on ImgNet32
OOD
OOD
OOD
OOD
OOD
IDF
223.0
3.60
4.18
3.94
Bit-Swap 23
44.9
3.97
3.00
3.87
4.23
HiLLoC 48
156.4
3.56
2.35
3.52
4.20
3.89
3,25
L3C 335
19.11
4.34
3.21
4.55
4.30
4.34
NeLLoC (r 0)
0.49
3.64
2.38
3.54
3.93
3.63
3.37
NeLLoC (r = 1)
1.34
3.56
2.26
3.47
3.85
3.55
3.31
NeLLoC 2.75
3.51
2.21
3.43*
3.82
3.53"
3.29
5 Conclusion and Future Work
In this work, we propose the local autoregressive model to study OOD generalization in probabilistic image modelling and establish an intriguing connection between diverse applications: OOD detection and lossless compression. We verify and then leveraged hypothesis regarding local features and generalization ability in order to design approaches towards solving these tasks. Future work will look to study the generalization ability of probabilistic models in other domains e.g. text or audio, order widen the benefits of the proposed OOD detectors and lossless compressors.
Acknowledgments
We would like to thank James Townsend and Tom Bird for discussion on the HiLLoC experiments. thank Ning Kang for discussion regarding the parallelizability of NeLLoC.
10

---

References
[1] Ahmed and A. Courville. Detecting semantic anomalies. In Proceedings of the AAAI Conference Artificial Intelligence, volume 34, pages 3154-3162, 2020.
[2] A. A. Alemi, I. Fischer, and J. V. Dillon. Uncertainty in the variational information bottleneck. arXiv preprint arXi:: 2018.
[3] R. v. d. Berg, A. A. Gritsenko, M. Dehghani, C. K. Sønderby, and T. Salimans. Idf++;: Analyzing and improving integer discrete flows for lossless compression. arXiv preprint arXiv:2006. 12459, 2020.
[4] T. Bird, F. H. Kingma, and D. Barber. Reducing the computational cost of deep generative models with binary neural networks. arXiv preprint arXiv:2010.13470 2020.
[5] C. M. Bishop. Novelty detection and neural network validation. IEE 1 Proceedings-Visionn Image and Signal processing, 141(4):217222, 1994.
[6] Boutell and T. Lane. Png (portable network graphies) specification version 1.0. Network Working Group, pages 1-102, 1997.
[7] R. Child, S. Gray, A. Radford, and I. Sutskever. Generating long sequences with sparse transformers. arXiv preprint arXiv:/904 10509, 2019.
[8] H. Choi, E. Jang, and A. A. Alemi. Waic, but why? generative ensembles for robust anomaly detection, arXiv preprint 1810.01 2018.
[9] Clanuwat, M. Iriza,, Kitamoto, A. Lamb, K. Yamamoto, and Ha. Deep learning for classical japanese literature, 2018,
[10] J. Deng, W. Dong, R. Socher, Li, K. Li, and L. Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248-255. Icee, 2009.
[11] Duda. Asymmetric numeral systems. arXiv preprint #xXiv:0902.0271. 2009.
[12] Duda, Asymmetric numeral systems: entropy coding combining speed of huffman coding with compression rate of arithmetic coding, arXiv preprint arXiv: 2013.
[13] F. Giesen. Interleaved entropy coders. arXiv preprint arXiv:1402.3392, 2014.
[14] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770- 778, 2016.
[15] D. Hendrycks and K. Gimpel. baseline for detecting misclassified and out-of-distribution examples in neural networks. arXiv preprint 1rXiv.1610.02136. 2016.
[16] D. Hendrycks, M. Mazeika, and T. Dietterich. Deep anomaly detection with outlier exposure. arXiv preprint a 2018.
[17] G. E. Hinton. Training products of experts by minimizing contrastive divergence. Neural computation, 14(8):1771-1800, 2002.
[18] Ho, E. Lohn, and P. Abbeel. Compression with flows via local bits-back coding. arXiv preprint arXiv: arXiv: 08500, 2019.
[19] E. Hoogeboom, J. W. Peters, R. d. Berg, and M. Welling. Integer discrete flows and lossless compression. arXiv preprint arXiv: 2019.
[20] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6000 19980 2014.
[21] D. Kingma and P. Dhariwal. Glow: Generative flow with invertible 1x1 convolutions. arXiv preprint 1rXiv:1807.03033 2018.

---

[22] D. P. Kingma and M. Welling. enco-ing variational bayes. arXiv preprint arXiv: arXiv: 1312.6114, 2013.
[23] Kingma, P. Abbeel, and J. Ho. Bit-swap: Recursive bits-back coding for lossless compression with hierarchical latent variables. In International Conference on Machine Learning, pages 3408-3417. PMLR, 2019.
[24] P. Kirichenko, P. Izmailov, and A. G. Wilson. Why normalizing flows fail to detect out-of- distribution data, arXiv preprint arXiv:2006.08544 2020.
[25] A. Krizhevsky, G. Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[26] R. Krusinga, S. Shah, M. Zwicker, T. Goldstein, and D. Jacobs. Understanding the (un) interpretability of natural image distributions using generative models. arXiv preprint arXiv:1901.01499 2019.
[27] B. M. Lake, R. Salakhutdinov, and J. B. Tenenbaum. Human-level concept learning through probabilistic program induction. Science, (50(62666:1332211333 2015.
[28] Y. LeCun. The mnist datahase of handwritten digits. http://yann. lecun. com/exdb/mnist// 1998.
[29] K. Lee, K. Lee, H. Lee, J. Shin. A simple unified framework for detecting out-of-distribution samples and adversarial attacks. In Advances in Neural Information Processing Systems, pages 7167-7177, 2018.
[30] L. Lian and W. Shilei. Webp: A new image compression format based on vp8 encoding. Microcontrollers & Embedded Systems, 3, 2012.
[31] Liang. Y. Li, and R. Srikant. Enhancing the reliability of in neural networks. arXiv preprint arXiv: 7066 770.026690 2017.
image detection
[32] Z. Liu, P. Luo, X. Wang, and X. Tang. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV), December 2015.
[33] D. J. MacKay. Information theory, inference and learning algorithms. Cambridge university press, 2003.
[34] A. Mahmood, J. Oliva, and M. Styner. Multiscale score matching for out-of-distribution detection, arXiv preprint arXiv:2010.13132 2020.
[35] F. Mentzer, E. Agustsson, M. Tschannen, R. Timofte, and L. V. Gool. Practical full resolution learned lossless image compression. In Proceedings the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 10629-10638 2019.
[36] E. Nalisnick, A. Matsukawa, Y. W. Teh, D. Gorur, and B. Lakshminarayanan, Do deep generative models know what they don't know? International Conference on Learning Representations, 2019.
[37] E. Nalisnick, A. Matsukawa, Y. W. Teh, and B. Lakshminarayanan. Detecting out-of-distribution inputs to deep generative models using a test for typicality. arXiv preprint arXiv: 2019.
5,
[38] Y. Netzer, T. Wang, A. Coates, A. Bissacco, B. Wu, and A. Y. Ng. Reading digits in natural images with unsupervised feature learning. 2011.
[39] M. Rabbani. Jpeg2000: Image compression fundamentals, standards and practice. Journal of Electronic Imaging, 1 112):286, 2002.
[40] J. Ren, J. Liu, E. Fertig. J. Snoek, R. Poplin, M. Depristo, J. Dillon, and B. Lakshminarayanan. Likelihood ratios for -ut-t--disttibution detection. In Advances in Neural Information Process- ing Systems, pages 14707-14718, 2019.
[41] T. Salimans, A. Karpathy, X. Chen, and D. P. Kingma. Pixelcnn++: Improving the pixelenn with discretized logistic mixture likelihood and other modifications. arXiv preprint arXiv: 1 2017.
12

---

[42] C. S. Sastry and S. Oore. Detecting out-of -itto---ditributionn examples with in in-istribution examples and gram matrices. arXiv preprint 1912::191..2233 2019.
[43] R. Schirrmeister, Zhou, Ball, and D. Zhang. Understanding anomaly detection with deep invertible networks through hierarchies of distributions and features. arXiv preprint arXiv:2006. 108428 2020.
[44] J. Serrå, D. Álvarez, V. Gómez, O. Slizovskaia, Núñez, and Luque. Input complexity and out-of-distributtin detection with likelihood-based generative models. arXiv preprint arXiv: 1909. 1480, 2019.
[45] C. E. Shannon. A mathematical theory of communication. ACM SIGMOBILE mobile computing and communications review, 5(1):3-55, 2001.
[46] Sneyers and Wuille. Flif: Free lossless image format based on maniac compression. In 2016 IEEE International Conference on Image Processing (ICIP), pages 66-70. IEEE, 2016.
[47] J. Townsend, T. Bird, and D. Barber. Practical lossless compression with latent variables using bits back coding. arXiv preprint 1XIi:19011.046666 2019.
[48] Townsend, T. Bird, Kunze, and D. Barber. Hilloc: Lossless image compression with 2019.
hierarchical latent variable models. arXiv preprint
[49] A. Van Oord, N. Kalchbrenner, and K. Kavukcuoglu. Pixel recurrent neural networks. In International Conference on Machine Learning, pages 1747-1756. PMLR, 2016.
[50] I. H. Witten, R. M. Neal, and J. G. Cleary. Arithmetic coding for data compression. Communi- cations of the ACM, 30(6):520-544 1987.
[51] H. Xiao, K. Rasul, and R. Vollgraf. Fashion-mnist: novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv:1708.07744 2017.
13


Reviews:
[{"review_id": "wu5R3og72Sk", "reviewer": "Reviewer_t8Bq", "summary": "The paper discussed the counterintuitive phenomenon in OOD detection using generative models, and proposed a new method called non-local feature density which fixed the issue and achieved the SOTA performance, because non-local features are shown to contain more semantic information. Further, the authors proposed an image compressor method, NeLLoC, based on the local features, because local features are shown to constitute a domain prior for all images so that NeLLoC is more generalizable to OOD images. ", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "- Originality: the authors provides a new method for fixing the issue of higher likelihood of OOD inputs in the generative model based OOD detection methods. The general idea of the proposed method is similar to the previous methods: the full generative models include both local (non-semantic) and non-local features (semantic). Local image features appear largely common across the real-world image manifold and can be treated as a domain-prior;  only non-local features which typically contain semantic image information, are useful for OOD detection. So the designed OOD score is the unnormalized likelihood of non-local features, i.e. p(x|full model)/p(x|local model).  \nCompared with the existing likelihood ratio methods [35] and [38,39], the proposed method is more direct to address the issue of common local features dominating the likelihood. But the proposed method seems to be specific to the PixelCNN model architecture. I wonder if the method can be generalized to other generative models.\n\n- Quality: the paper is well written, and the results are clearly presented. \n\n- Clarity: It would be great if the algorithm for NeLLoC can be presented, describing the exact computational steps for compression or decompression.\n\n- Significance: The proposed method is specifically designed based on the PixelCNN model. It would be great if the method can be generally applied to other types of generative models. \n\n- Minor comments:\n- Figure 3 (a) shows a slight overlap between the IND and OOD distributions, but Table 3 shows the AUROC for the same experiment is 1.00.\n- Typo: eq(3), k->d\n\n\n*** Update: Thank the authors for your responses. It is great that the framework can be applied to other autoregressive models such as Transformer. I wonder if Transformer model can achieve even better performance. After reading other reviews and responses, I decided to keep my score. ***", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors adequately addressed the limitations and potential negative societal impact of their work.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "wQzvMLdzCLb", "reviewer": "Reviewer_kxNK", "summary": "A surprising observation in the OOD literature is that OOD detection based on density estimation fails on some image datasets since the learned density tends to be higher on OOD data. A hypothesis for this phenomenon is that the likelihood is dominated by the local features of the image, resulting in the high-level features being ignored. Motivated by this observation, this paper proposes a model that learns local features of images. This model is used for two different applications: (1) by combining it with a full density estimation model, it gives a better OOD detection method, achieving state-of-the art performance (2) by pairing the model with the artihmetic coding algorithm, it gives a dataset compression method that achieves state-of-the-art compression rates on in- or out-of-distribution datasets.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Both the ideas of utilizing the problematic and puzzling phenomenon of failures of density-based OOD detectors for a different approach (compression) and likelihood-ratio of the full and local model are novel and valuable. The reported results are impressive, making this method a promising approach for both these tasks. In the compression task, being able to achieve such results with way smaller models is also impressive.\n\nQuestions/comments:\n\n- I am wondering how the runtime of different methods compare. The paper does comment on how the runtime of their proposed method scales and regarding its parallelizability, but an additional table of results showing the exact time for compression/decompression is useful.\n\n- Why are many entries in table 3 missing? (although, they are irrelevant for all but the last column, since the AUROC of the proposed method 1)\n\n---\nPost-author response update: I appreciate authors taking the time to respond and provide table of run-times. I have read other reviews and responses and decided to keep my score the same.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I cannot think of limitations beyond what is mentioned in the paper.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "7"}, "extra_text": {}}, {"review_id": "T0G702LVtgH", "reviewer": "Reviewer_XCL4", "summary": "Summary: This paper study on applying generative models on unsupervised out-of-distribution detection. Authors hypothesize that local features common to both in-distribution and out-of-distribution data dominate the likelihood of generative models. The authors show evidence of the hypothesis via comparing the fully autoregressive model’s likelihood and autoregressive model with local dependencies. Motivated by the results, the authors propose a ratio between the likelihood of model trained with full-dependencies and the likelihood of model trained with local dependencies. The authors show that the proposed metric outperforms conventional generative-model-based methods on CIFAR-10 and FashionMNIST domains. Finally, the authors apply local autoregressive models on universal lossless compression. The proposed compression scheme outperforms conventional methods.\n", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "UPDATE : \nI acknowledge that I have read the author response as well as the other reviews.\nAlthough authors replied my concerns on the novelty, I still believe the main intuition of the paper, \"directly model the in-distribution dataset, using only local feature information\" is straighforward application from the idea of previous works. \nDue to the previous reasoning, I insist deleting section 3.2 and Figure 1. \nHowever, I understand the virtue of the method on unsupervised novelty detection, which doesn't need specific augmentations.\nFurthermore, I appreciate the clarifications on lossless compression.\nTherefore, I am changing my score to 5 and confidence to 4.\n\n=========================\n\nUPDATE (2):\n\nWhile this is relatively minor, I question the following points in the reply.\n\nOur major goal of OOD detection experimental work is to show that our simple and principled approach can perform comparatively strongly. We believe our experimental results strongly support this claim. At the time of submission, we are not aware of any data augmentation free, model ensemble free methods, capable of consistently outperforming our approach.\n\n=> Consider looking at the paper \"Multiscale score matching for out of distribution detection\" [https://arxiv.org/pdf/2010.13132.pdf]\n\nWhile the experiments are performed only on CIFAR-10 vs SVHN, the stark contrast on the performance reasons me whether the scheme proposed by the authors is competitive, or just a method that can be outperformed by a better generative model.\n===========================================\n\nUpdate (3)\n\nI acknowledge that I read the response from the authors as well as the other reviews. I believe that the methodology proposed in the paper shows State-of-the-art performance among augmentation-free unsupervised out-of-distribution detection methods. However, I'm still on the side that the main novelty of the paper is relatively minor. Therefore, I still stick to the current ratings. \n\n==========================================\n\nPros\n-       Application on the universal lossless compression task is interesting, while this is not the first work to apply generative models on lossless compression. Furthermore, the proposed method outperforms conventional methods\nCons \n\n-       My deepest concern on the paper is that the proposed hypothesis is not new and published in NeurIPS 2020. We can find in [1] where the authors verified that GLOW and RealNVP encode representations overfitted on local pixel correlations. I would not be surprised that the same thing happens to PixelCNN since they all failed in unsupervised anomaly detection [2] and they both exhibit an encoder-decoder-like structure. As a result, this paper sounds like an extension of existing ideas to a different generative model. \n \n-       Experiment results are limited. In table 3, the authors compare their proposed metric against diverse OOD detection methods. However, the authors compared their results against comparably weak baselines. I can find other published practices [3] [4] in unsupervised novelty detection that outperform the result. \n\nOverall, I think the main idea of this paper is nothing different from the idea of [1], and the experimental results are not compelling or convincing compared to recently published unsupervised novelty detection methods [4]. Therefore, I am leaning towards rejecting the paper.\n\nComments\n\n-       Authors should cite [1].\n\n-       I think FashionMNIST is comparably too “easy” dataset for verifying out-of-distribution detection performance. I suggest experimenting on diverse in-distribution datasets, e.g., TinyImageNet, CelebA, LSUN. Furthermore, there are too many blanks in the experiment result that it is hard to convince that the proposed metric improves over conventional methods. Furthermore, it would be beneficial to compare the performance of the paper against more diverse unsupervised OOD detection methods.\n\n-       I would like to see the effect of variation in local horizontal length(l) and kernel size(k) as means of OOD detection performance across different in-distribution datasets. It would be beneficial to show whether the optimized hyperparameters can be applied in other unobserved datasets.\n\n\nReferences.   \n[1] Why normalizing flows fail to detect out-of-distribution data. Polina Kirichenko∗ , Pavel Izmailov∗ , Andrew Gordon Wilson\n\n[2] Do deep generative models know what they don’t know? Eric Nalisnick et al.\n\n[3] Novelty detection via blurring. Sungik Choi and Sae-Young Chung\n\n[4] CSI: Novelty Detection via Contrastive Learning on Distributionally Shifted Instance. Jihoon tack, Sangwoo mo, Jongheon Jeong, and Jinwoo Shin.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "While authors did not focus on explaining the potential negative social impact of their work, I believe that is not such an issue.", "review_text": "", "extra_scores": {"ethical_concerns": "I could not find any ethical issues on the paper. ", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "Qs5ql60zY6W", "reviewer": "Reviewer_t6SW", "summary": "This paper proposes a new probabilistic image modeling approach based on a local autoregressive model. The local autoregressive model is implemented like the masked convolution operation in the pixelCNN model. The model is applied to two downstream tasks of OOD detection and lossless image compression. The OOD score is computed using the ratio between the likelihood from the full model versus the one from the local model. The idea is simple but shows some improvement over previous models. \n\n", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "[Update] I have read the author responses and comments by the other reviewers. My previous concerns about clarity and significance have been partly resolved, so I'm changing my score from 5 to 6. \n--------------------------------------------------------------------- \n\nThe main idea comes from the intuition that the local features are shared between different image datasets and dominate the generalization ability, which is interesting. \n\nIn terms of clarity, it was quite hard to follow the main idea of the paper at first, probably because it is not clearly introduced how the core model is connected to the two downstream tasks which seem quite unrelated. Moreover, the target tasks are explained in sections 2, 4, and 5 while the main component of the proposed approach is explained in sec 3. Also, the methods and results are mixed in the same section.  For better readability, the overall structure needs to be improved, for example, the authors may provide more introduction about the overall flow of the paper in the beginning and rearrange the following materials. \n\nIt is not clear from Table 3  if the performance gain over the previous methods for OOD detection is significant. Actually, it is not clear where the main focus of this paper is.  It might be better to choose one target task (probably OOD detection) and put more emphasis on that by investigating the performance behavior in more details. \n\nAs the author argues, time complexity is an important factor in the case of lossless compression.  I wonder how the actual encoding and decoding time compares with those of other methods. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "There is no explicit discussion on the limitations or societal impact of the proposed work, but there appears to be no direct negative societal impact from this work. It will be helpful if more discussion on the limitations of the proposed method is included. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The main concerns about this work shared by the reviewers were around novelty and presentation. One reviewer felt that a key idea underlying the work has already featured in several other works in recent years, and thus the novelty of the work is limited. I am inclined to agree with this, however, I believe this work could be a useful reference to help more people understand the intricacies of OOD detection and the biases of likelihood-based models (especially autoregressive models), and the practical demonstration that a tiny local model is all you need for efficient lossless image compression, is valuable in itself.

Therefore, I have decided to recommend acceptance. This is of course conditional on the authors including in the manuscript all the additional results they have provided, and reworking the parts that were deemed unclear as set out in the authors' responses.
Decision: Accept (Poster)