Self-supervision through Random Segments with Autoregressive Coding (RandSAC)
Authors: Tianyu Hua
Yonglong Tian
Sucheng Ren
Michalis Raptis
Hang Zhao
Leonid Sigal
Venue: ICLR 2023 poster (2023)

Abstract: Inspired by the success of self-supervised autoregressive representation learning in natural language (GPT and its variants), and advances in recent visual architecture design with Vision Transformers (ViTs), in this paper, we explore the effects various design choices have on the success of applying such training strategies for visual feature learning. Specifically, we introduce a novel strategy that we call Random Segments with Autoregressive Coding (RandSAC). In RandSAC, we group patch representations (image tokens) into hierarchically arranged segments; within each segment, tokens are predicted in parallel, similar to BERT, while across segment predictions are sequential, similar to GPT. We illustrate that randomized serialization of the segments significantly improves the performance and results in distribution over spatially-long (across-segments) and -short (within-segment) predictions which are effective for feature learning. We illustrate the pertinence of these design choices and explore alternatives on a number of datasets (e.g., CIFAR10, ImageNet). While our pre-training strategy works with vanilla Transformer, we also propose a conceptually simple, but highly effective, addition to the decoder that allows learnable skip-connections to encoder feature layers, which further improves the performance.

SELF-SUPERVISION THROUGH RANDOM SEGMENTS WITH AUTOREGRESSIVE CODING (RANDSAC)
Tianyu Hua Yonglong Tian
Sucheng Ren
Michalis Raptis Hang Zhao²
Leonid Sigal
l University of British Columbia
2 Massachusetts Institute of Technology
South China University of Technology
*Google Research CCnada CIFAR AI Chair
TTsinghua University
Veector Institute for AI
ABSTRACT
Inspired by the success of self-supervised autoregressive representation learning in natural language (GPT and its variants), and advances in recent visual architec- ture design with Vision Transformers (ViT'), in this paper, we explore the effect various design choices have the success of applying such training strategies for visual feature learning. Specifically, we introduce a novel strategy that we call Random Segments with Autoregressive Coding (RandSAC). In RandSAC, we group patch representations (image tokens) into hierarchically arranged seg- ments; within each segment, tokens are predicted in parallel, similar to BERT, while across segment predictions are sequential, similar to GPT. We illustrate that randomized serialization of the segments significantly improves the perfor- mance and results in distribution over spatially-long (across-segments) and short (within-segeh sethinn-eggmnt) predictions which are effective for feature learning. We illus- trate the pertinence of these design choices and explore alternatives on number of datasets (e.g., CIFARIO, CIFAR100, ImageNet). While our pre-training strat- egy works with vanilla Transformer,
also propose conceptually simple, but highly effective, addition to the decoder that allows learnable skip-connections to encoder's feature layers, which further improves the performance.
INTRODUCTION
Deep learning has powered enormous successes Computer Vision and NI.P over the past 10, or so, years. It has lead to significant improvements in object detection (Redmon et al., 2016), segmenta- tion (He et al., 2017), as well as higher-level cognition tasks (e.g., Visual Question Answering (Antol et al., 2015), Visual Navigation (Mayo et al., 2021), etc.). These successes have been enabled by both advances in parallel hardware (GPUs) and, perhaps more importantly, large-scale ssecispe labeled datasets that allow supervised learning. This appetite for large data has, until very recently, stagnated progress, particularly in building general-purpont visual architectures.
These types of considerations date back to the early days of machine learning, and deep learning in particular, where has long been postulated that unsupervised, or self-supervised, learning could allow learning of robust and general feature representations that can then be readily used (or fine- tuned) to target tasks. Self-supervised learning has been explored in computer vision in various forms: denoising autoencoders (Pathak al., 2016; Vincent al., 2008), colorization (Zhang al., 2016) or jigsaw puzzle (Doersch et al., 2015; Noroozi & Favaro, 2016) proxy objectives. However, the success of such self-supervised pre-training was somewhat limited. In contrast, the success of similar self-supervised ideas in NL.P has been much more dominant with GPT (Brown et al., 2020) and BERT (Devlin et al., 2018) architectures, and their variants. These pre-training strategies now enable ofte-offthhear performance
wide array of natural language tasks.
Recent advances in vision architectures, such as Vision Transformers (ViT) (Dosovitskiy et al., 2021; Liu et al., 2021), which serialize visual 2d data, have opened an opportunity to apply similar large scale pre-training techniques in vision, with increasing successes. Self-supervised pre-training techniques with ViTs be characterized into two broad categories: contrastive and predictive; as well as their combinations. In contrastive learning, pre-training architectures are learned to be invariant to certain perturbations in data (e.g., spatial shifts, color jitter) by forming positive and

---

Random Segments with Autoregressive Coding
Figure I: Randomized Autoregressive Segment Prediction. Illustration of our autoregressive seg- ment prediction framework (RandSAC) RandSAC breaks the image into tokens which are arranged into segments (here squares of size 2). The autoregressive (GPT-style) transformer- is then trained to predict segments in a randomly sampled serialization order. As a result, tokens within segments are predicted in parallel, while segments themselves are predicted sequentially.
model
negative pairings of augmented data samples. This is a powerful technique, but requires design- make assumptions about invariances that the architecture should learn. In addition, purely contrastive models tend to incorporate center bias (Chen et al., 2022; 2021a), which makes them less transferable for tasks such as segmentation where non-object centric regions need be mod- eled. Alternatively, predictive models learn to predict elements of the scene, either in parallel by reconstructing masked regions/tokens (Bao et al., 2022; He et al., 2021) (a.k.a,, masked image mod- cling or BERT-style pre-training) or to predict images in auto-regressive language-modelinng observe that on the NLP
manner
(Chen al., 2020a) (a.k.a., GPT-style pre-training). It is interesting side, GPT models have shown to be powerful, while vision models have gravitated more towards BERT-style pre-training both with visual (Chen et al., 2020a; Bao et al., 2022) and multi-modal data (Lu et al., 2019; Su et al., 2020).
Motivated by this, we adopt an autoregressive pre-training strategy (see Figure 1) and ask a number of important empirical questions about the use of such pre-training and what makes it effective. Specifically, (I) we ask what granularity (scale) and shape of tokens (patches, blobs) is most effective and how it affects the performance?
(2) How best to serialize predictions? For example, previous approaches, such as image GPT (Chen et al., 2020a), leveraged raster ordering. While such ordering is perhaps "optimal" from correlation and predictive/generatie (van den Oord et al., 2016) points of view, we show that is not optimal for general feature learning. We also explore (3) whether deterministic vs. stochastic tokenization and serialization are helpful. Finally, (4) we explore the effective interactions between the decoder and encoder layers; proposing a new ViT architecture that uses learned skip connections between encoder and decoder layers to improve performance.
Contributions. We make two core contributions. First, we propose a new pre-training strategy that leverages (randomly) sampled hierarchical segment cluster traversals autoregresively train ViT models. This allows both short- and long-term spatial predictions, allowing distribution over easy and hard predictive tasks'. We note that the effectiveness of single random segment inpainting was initially observed in (Pathak et al., 2016). but is notably missing from most recent self-supervised strategies. Our pre-training strategy generalizes this observation and strategy to hierarchical and serialized predictions. Second, we propose a flexible ViT decoder that at each decoding layer learns to dynamically attend over different levels features in the encoder. This in effect creates learned ciipcconnections, compared UNet (Ronneberger et al., 2015) and others that require fixed connections in a symmetric encoder-decoder design, which further improve the performance.
Discussion.
The above pre-training strategy, while empirically motivated, is also loosely modeled after human vision. Humans attend to the scene by a sequence of foveal observations, where an eye shifts over series of fixation points; such motions range and voluntary, while others are local and involuntary (a.k.a,, microsaccades (Rolfs, 2009)). Our segments can be "viewed" as predictive foveal regions, and the hierarchical serialization of such regions as the combination of micro and macro saccades. The significant difference from human vision, is that in human vision saccades are purposeful and have been shown to be conditioned on the task (Yarbus, 1967). In contrast, our pre-training such "saccadic" movements are randomly sampled.
called succades. Some saccades are long-
'This is, in part, motivated by (He et al., 2021) which observe that in BERT-style pre-training high amount of masking (as much 75%), which corresponds to harder predictive tasks, leads to better feature learning.
2

---

Learning& a purposeful policy for hierarchical serialization of segments, would be an interesting future work. However, this is a difficult task that is beyond the scope of this paper.
RELATED WORK
Transformer- Natural Language Modeling. In the field of natural language processing (NLP), two dominant self-supervised language modeling paradigms are Masked Language Mod- cling, such as BERT (Devlin et al., 2018), and GPT-style autoregressive pre-traie pre-training (Brown et al., 2020; Radford & Narasimhan, 2018; Radford et al., 2019). Given a sentence, BERT and its variants (Lan et al., 2020; Liu et al., 2019) pre-train transformer encoders by predicting randomly masked out input words, referred to tokens. Such frameworks model the bidirectional (contextual) de- pendencies between the visible tokens and the corrupted/masked tokens. GPT, which can be viewed as a special case of the transformer decoder, on the other hand, models the left-to-right natural order of languages. Recent advances in large-scale generative language modeling show powerful few-shot capabilities and are believed to be a promising path towards general machine intelligence. Permutation- based autoregressive model (Yang et al., 2019) was proposed to bridge the gap be- tween autoregressive language modeling and masked autoencoding by maximizing the likelihood over all permutations of the factorization order. We take inspiration from GPT--ttyle autoregressive pre-training in formulating our model, and focus on important aspects of mapping such strategy onto visual (ViT) models, where tokenization and serialization are not as well defined as in language.
Contrastive Image Learning. Contrastive methods (Chen et al., 2020b; He et al., 2020; van den Oord et al., 2018; Tian et al., 2020) and their serpativ-sample-fieee variants (Chen & He, 2021; Grill et al., 2020; Hua et al., 2021; Zbontar et al., 2021) have emerged as a dominant research direction for wttervvieeesssefssseeriieeeeseee visual representation learning over the past 1-2 years. By building agreement among augmented versions of the input data, image features that are invariant of those perturbations can be learned. This method implicitly assumes set of representational invariance (e.g., color and spatial invariance). Once such representations are learned they are either used di- rectly, or fine-tuned, to one or more downstream supervised tasks (e.g. classification, detection, segmentation). When a downstream task violates the aforementioned invariance assumptions, they display poor transferability (Xiao al., 2021). For example, the center-bias (Chen et al., 2022) and small-object feature suppression (Chen et al., a) have been observed in prior works. Masked image modeling & autoregressive image encoding, of which our method is an instance, tend to perform better in such circumstances (Bao et al., 2022; He et al., 2021).
Masked Image Modeling. Early CNN-based masked image modeling, also known as image in- painting (Doersch et al., 2015; Pathak et al., 2016; Yu et al., 2018), has shown promising results but failed to become a predominant training paradigm, in part, due to its inferior performance with respect to large-scall supervised pre-training (e.g., on ImageNet). The recent trend of incorporating transformers into vision architectures (Carion et al., 2020), or replacing CNN completely (Doso- vitskiy al., 2021), by tokenizing images into a grid of non-overlapping patches, have enabled application of large scale NI.P pretraining techniques in vision, e-g., (Bao et al., 2022; He et al., 2021; Wei et al., 2021; Xie al., 2022). Directly applying them to image pixels, however, leads to inferior performance (Chen et al., 2020a; Dosovitskiy et al., 2021). To this end, BE (Bao et al., 2022) proposes to predict discrete masked image tokens. Masked Autoencoder (MAE) (He et al., 2021) suggests 75% random masking ratio for image modeling: and SimMIM (Xie et al., 2022) studies different masking strategies for pretraining. MaskFeat (Wei et al., 2021) investigates five dif- ferent reconstruction targets. TinyMIM (Ren et al., 2023) introduces distilling token relations from MAE model, and SplitMask (EI-Nouby et al., 2021) illustrates the ability of BETT to train with small scale pre- training datasets. Our proposed RandSAC strategy, is related to masked image modeling, but is autoregressive in nature.
Autoregressive Image Encoding, Compared with BERT-style pre-training for vision transformers, GPT-like autoregressive models have been overlooked due to their complexity introduced by dense image pixels. In image GPT (Chen al., 2020a), images are limited to 64 x 64 4096 pixels. The 4096 pixels are tokenized and serialized in raster-order before feeding into a causal transformer. The quadratic time/space complexity of self-attention prevents the scaling of such approaches.
3 RANDOM SEGMENT WITH AUTOREGRESSIVE CODING
RandSAC learns representations through autoregressive image segment prediction. It partitions tokenized image into random spatially coherent non-overlapping (hierarchical) segments, serializes them, and then autoregressively predicts tokens within these ordered segments. As result, the token
3

---

predictions between segments are sequential, while within segment are parallel. This training strategy has four important components that we will explore:
Tokenization. To use a transformer-bant based architecture, images need to be tokenized, i.e. transformed into set of basic image elements. For example, some approaches discretize images (Chen et al., 2020a), while others patchify them (Cordonnier et al., 2020; Bao al., 2022; Dosovitskiy al., 2021; He et al., 2021: Xie et al., 2022). Tokenization strategy dictates the scale and number of tokens, which affects performance and computation cost.
Segment Partitioning. After tokenizing the image, the tokens are grouped into spatially coherent segments. Those segments are autoregressively predicted following some pre- scribed serialization order. The size and shape of segments and the way they are traversed can affect training and downstream performance.
Serialization Strategy. Serialization strategy affects the traversal order of segments. In prior autoregressive modeling (Chen al., 2020a) raster-order assumed. We show that stochastic (i.e., randomized) serialization is much more effective.
Transformer Architecture. In a GPT-style autoregressive model, the target sequence is identical to the shifted input sequence throughout training. However, for random segment prediction, the target sequence order varies for each sample. To enable this, we leverage transformer decoder which takes as input position of each token and outputs its predicted representation conditioned on the transformer encoded context. In addition, we propose novel trainable skip-connection layer for efficient decoding.
In the following section, the default option for model architecture the vanilla masked transformer introduced in Section 4. We experiment with two different datasets, CIFARIO (Krizhevsky, 2009) and, where appropriate, ImageNet100 (Tian et al., 2020). Evaluation protocols are described in Section 5, and implementation details are in the Supplemental. We use a simple mean square error (MSE) as our pixel reconstruction objective.
3.1 FROM PIXELS TO TOKENS
Tokenization. We start from raster-order serialization and compare two different tokenization strate- gies introduced by iGPT (Chen et al., 2020a) and ViT (Dosovitskiy et al., 2021). Assume a dataset ' of images X Fx where H, W, C are the height, width, and the number of channels of the image. We reshape each image into N = HW/PP p2 patches, where P is the resolution of each patch. Tokens obtained by linearly projecting the patches X [x,);= and serialized row-by-row.
For pixel prediction experiment, we set P p= 1, letting image patch size be 1 x pixels (see Figure (b)). For ViT style patch prediction experiment, we split the 32 x 32 CIFARIO image into 8 64 patches (see Figure 2 (c)), each patch consists of 4 x 4 pixels (P = 4). Note that for a fair comparison, we didn't strictly follow iGPT, where they minimize the negative log- likelihoooo of the quantized RGB values. We simply adopt mean squared error (MSE) between the predicted and target pixel values for all our experiments following (He al., 2021). Note that for visualizations in Figure 2 we use downsampled CIFARIO image.
The results for these two tokenization options
pixel-raster
patch-raster
illustrated in Table 1 (additional scales are in Suppicmental) under pixel-raste and match-raster respectively in terms of linear probing and fine-tuning accuracy (see Sec. 5.1 for definition of metrics). From the point of view of representation learning, patches are substantially better. Further, computationally. transformer uses O(n²) in both time and space with respect the
LIN(1) FT())
41.70
55.53
59.35
78.67
Table 1: Tokenization on CIFAR1O.
the self-attention mechanism sequence length. Hence for pixel tokenization, the complexity is O(HW)))) For patches, the com- plexity is reduced ((HW/P22)) In our CIFARIO experiment, when P 4, the complexity of training is lowered by a factor of pt 256. Hence, patches result in better tokenization.
Stochastic Serialization. Randomized pretext tasks play an important role in a range of self- supervised learning algorithms. In NLP, for example, (Yang et al., 2019) improves fxxe-order autoregressive language models by allowing all possible permutations of the factorization order
4

---

(a)
(b)
(c)
(d)
(e)
Figure 2: Autoregressive Prediction Schemes. Left-to-right: (a) original image from CIFAR 10; (b) raster-order pixel prediction; (c) raster-order patch prediction: (d) stochastic patch prediction; (e)
stochastic square segment prediction (M = 2); (f) stochastic blob segment prediction (K 5).
during training. For autoregressive ViT train- ing of stochastic token serialization, we adopt similar strategy by shuffling the token sequence for each image sample. Note that this does not mean that our prediction sequence is "or- derless". By moving from fixed raster-order prediction to randomized sequence prediction, keeping all else the same, observe 20% im- provement in linear evaluation and ~10% in fine-tuning (Table 2 CIFAR10). Improvements ImageNet100 more modest (3.67% and ~2% respectively). but still significant and overall stochastic serialization is clearly superior.
patch-raster
patch-random
CFIO-LIN(1)
55.53
75.53
CF10-FT(1)
78.67
87.52
IN100-LINC) IN100-FT())
49.35
53.02
82.13
84.15
Table 2: Serialization on CIFAR10 and Ima- geNet100.
3.2 GROUPING TOKENS INTO SEGMENTS
In this section, we introduce concept of segments, which we define as groups (or clusters) of tokens. Effectively each segment forms an equivalency class within our serialized order, where tokens are encoded and decoded in parallel. Across segments, however, predictions are still strictly sequential. The motivation for introducing segments two-fold. First, it allows us reduce the overall number of autoregressive prediction steps. Second, it allows our autoregressive strategy to effectively leverage aspects parallel, BERT-style, prediction locally. The autoregressive prediction steps can also be changed without introducing parallel prediction, simply by changing the patch size P. This is ineffective, however, as we show in Supplemental Section A.I. In what follows, we experiment with two spatially coherent segment strategies (square and blob) and then look at the importance of this spatial coherence in segment formation.
W. Square Segments. Once have grid of N patches of size x p. we reshape the tokens into set of square segments M x M, where the denotes the size of the square. The segment
count K of an image of H x W is thus de- our CIFARIO experiment, an input image of size 32 32 is tokenized into grid of 8 x tokens, each which is a x pixel patch. We set the square size M 2. The tokens are then split into (8/2)² = 16 segments, which are shuffled randomly for autoregressive prediction square segment size (M) in Table 3. Since the grid size is 8 x 8 for CIFARIO, we chose square sizes M 2, 4]. Note that, when M 8, there will be only one segment (e.g. K 1) and no in Table 2).
Square size M
1
HXW fined by: K (PX For example,
LIN(1)
75.53 87.52
81.38
79.38 90.23
FT()
91.38
Table 3: Square-random serialization as func- tion of on CIFARIO.
before. We list the representation quality with different
prediction can be made; M = is equivalent to no segments (ie., paacppataaaaae
Blob Segments. We define blob segments irregular elliptical segments defined by sampled Mixture of Gaussians. To obtain K random blobs for given image, we first sample K Gaussians with a range of means and standard deviations token x; which is at position (I;, y:) to the closest mixture component using Mahalanobis distance. We illustrate the square and blob strategies in Figure 2 (e) and (f), respectively. Note that beyond the shape, blob segments allow for variability in size squares do not. See details in Suppl. Section A.2.
the image space. Then we simply assign each
Analysis. As can seen from Table 4, both square segments and blob segments surpass segment-
5

---

free patch-based autoregression (see square-randon and blob-random compared with patch-random) 'Tee blob segments and square segments behave similarly. In addition, with blobs, we can easily mod- ify the number of segments. However, with squares, the segment number is constrained by the token number. A grid of 8 8 tokens can either be segmented into 4 or squares. grid size 13 13 can not be divided into any kind of squares. Blob segments, on the other hand, are more flexible.
patch-random
square-random
blob-random
CFIO-LIN(1)
75.53
81.38
82.52
CF10-FT(1)
87.52
91.38
91.53
IN100-LIN(1)
53.02
64.78
65.00
IN100-FT(1)
84.15
86.22
85.16
Table 4: Segments CIFARIO.
Do segments need to be spatially coherent? The idea of "segment" puts emphasis on the spatial coher- ence of the tokens. The upper part of Table 5 shows the performance of feature representations with respect to the number of blob segments K. In the bottom, we randomly shuffle all tokens so that tokens in any given "segment" longer spatially coher- ent. We observe that feature learning
Table 5: Segment Coherence, Representation quality with different number of segments. Below we randomly permute the segments such that their spatial coherence is disrupted.
Segment K
5
7
9
11
LIN(T)
80.87
81.82
82.52
$1.88
82.02
FT(T)
90.77
90.88
91.14
91.53
91.24
Shuffle
3
5
7
9
11
LIN())
76.73
77.73
76.69
78.59
76.99
FT(1)
89.63
89.75
89.22
90.00
89.15
deteriorates when segments are not spatially coherent. Note that segments without spatial coherence are still consistently better than
from Table 4.
3.3 HIERARCHICAL SEGMENT SERIALIZATION
Images are hierarchical: visual region of image can often be interpreted component of greater whole (Hinton, 2021) (e.g., parts make up object, object scenes, and so on). Such composi- tionality motivates hicrarchical groupings. our case of random segment serialization, we postulate that similar hierarchical traversal order, which adds certain degree of locality, may be useful.
In Figure we illustrate this concept that we operationalize. An image is first partitioned into 16 square segments, indicated by different colors and shades. We then group these 16 segments into larger partitions following the same logic for segment generation. Different colors (e.g.. blue, orange, purple, and green) represent these partition groups; segments that share partition differ in shade. Hierarchical serialization is obtained by randomly, and sequentially, predicting the segments inside of each partition group (shown by the black arrows), and then jumping to another partition group at random. Note that the segment-level (local) and partition-Ieve (global) serializations are both random. This idea can be extended to deeper hierarchies, with the depth of the hierarchy and grouping chosen based on the resolution and nature of the dataset.
Experimental results that compare flat serialization two-level hierarchy are illustrated in Table 6. We perform these experiments on both CIFARIO 10 and ImageNet100 datasets. Our experiments show that hierarchical serialization and prediction consistently outperform the flat counterparts.
X L
Flat Serialization (Square)
Hicranchical Scriafication (Square)
Serialization (Blob)
Hicranchical Scrialization (Blob)
Figure 3: Hierarchical Segment Serialization. We partition an image into hierarchy of segments (segments are illustrated by color and tokens within segment by shade). Autoregressive prediction done by following traversal of randomly generated hierarchical partitions.

---

Table 6: Hierarchical Segment Prediction. The number on top indicates the number of segments
K (e.g.. 4, 16) fla//no-hierarchy; the 16 + 4 indicates hierarchical variants with two levels 16 segments grouped into 4 partitions. Left/square and middle/blob results correspond Fig. 3.
Segments (square)
Segments (Blob)
CF10 Linear (1)
79.38
81.38
82.46
CF10 Linear (1)
80.87
82.52
82.71
81.09
80.97
82.61
CF1O Fine tune
89.61
91.38
91.66
CT10 Fine lune
90.77
91.14
91.20
90.63
90.57
91.15
IN100 Linear
55.88
64.90
Linear
64.92
IN 100 Fine- tune (1)
78.81
85.32
85.55
IN100 Fine tune (f)
81.34
84.36
84,48
83.07
86.06
86.18
4 ARCHITECTURE
Image GPT (Chen et al., 2020a) performs autoregressive prediction by shifting the source sequence one pixel to the right. Since the raster ordering of iGPT is fixed for all samples, the position for the next target token implicitly modeled by the transformer. In contrast, in RandSAC, the next token depends Moreover, when predicting the next segment, the tokens within each segment should be predicted jointly (in parallel). This requires lateral pathways that allow communication within target segments. To tackle the aforementioned problems, we propose to utilize the transformer decoder.
the serialization strategy, thus can vary from sample sample during training.
4.1 MASKED TRANSFORMER FOR SEGMENT PREDICTION
standard transformer has an encoder-decoder structure (Vaswani al., 2017). The encoder of a transformer maps a list of tokens X (X1, x,,) to sequence of hidden representations Z (Zi...., 2n). also known as the memory. Given X and source sequence X = (x1.... Xn-1). during training, the decoder masks the internal attention matrix with a causal mask and predicts the target sequence Xtgt = (x2,...,X, xn) autoregressively. Each layer of the transformer encoder has two sub-layers: multi-head self-attention and fully connected feed-forward network; both have residual connections. The decoder layer has a third attention sub-layer, which performs multi-head attention from the hidden representation Z to the target representation Xtat- We leverage attention masking to achieve autoregressive segment prediction using this framework; we discuss details next.
Autoregressive Segment Encoder. Figure shows our transformer encoder block and decoder block. We leave out the fully connected layer and residual connections for simplicity and only show the attentions. this visualization, there are six patches. These six patches are then grouped into three segments denoted by colors: green, blue, and red. The random segment serialization order is green + blue > red. One layer of transformer encoder is illustrated on the left in light green. Serialized six patches/tokens with added fixed sine-cosine positional encoding the input to the encoder. The encoder attention is masked following the serialized segment order: segments can attend themselves and preceding segments only. They are restricted from looking future seg- ments using the, illustrated, source mask. Lastly, since the last segment does not have succeeding segment, we only input the first four patches and leave out the two patches in the last segment.
Astonognaaive Sognent Profiction
Target Querles
Targel Mask
Enoooder JAVe
Monory
Meroryy
Input
Tanget
Figure 4: Attention-masking for Autoregressive Segment Prediction. For image converted into a sequence of patches, we adopt a masked encoder-decods transformer (Vaswani et al., 2017) for autoregressive segment prediction. In the encoder, causal source mask enables given segment to only attend over preceding segments and the tokens within itself. The decoder, given the position of tokens (i.e., target queries), predicts tokens within each segment conditioned on encoded previous segments (enabled by the memory mask).
7

---

Autoregressive Segment Decoder. The imput for the transformer decoder, illustrated on the right of Figure in pink, is a set of fixed positional encodings to guide the reconstruction of target segments and tokens. Similar to the encoder imput where leave out the last segment patches, in the decoder, we shift the target sequence one segment to the left and ignore the positional encodings of the first segment because it does not have preceding segment. The self-attention layer of the decoder is masked the same way as the encoder for autoregressive segment decoding. This layer enables co-attention over preceding and current segments for context.
Evaluation. During linear evaluation and fine-tuning, both the attention masks and decoder are removed, and the encoder used as feature extractor for the downstream supervised classification.
4.2 TRAINABLE SKIP CONNECTIONS
The original transformer decoder layer can only attend to the same encoder output, often from the last layer the encoder. In contrast, CNN encoder-decoder architectures are often symmetric with skip connections between encoder and decoder layers, e.g. UNet (Ronneberger et al., 2015). We hypothesize that our design, skip-connections between transformer encoder and decoder can sim- ilarly be beneficial. To enable such skip connections, propose trainable skip connection module that learns how to assign encoder memory to the decoder layers. Specifically, for transformer with Treee and T=doe number of layers,
learn linear layer with parameters W E Rhone Ldee, such that: Z' = Ehene Wi,: H H where H an encoder representation from layer & and Z' the formed memory for decoder layer 1. Note, the linearly formed memory cells are conditioned on, and different, for each individual decoder layer. We refer the reader the Supplemental Section A.3 for details and experiments that validate the effectiveness of this design and discuss efficiency.
EXPERIMENTS
We test RandSAC in two drastically different settings: low-data and ImageNet-IK pre-training. We evaluate the classification performance of our pretrained backbone with linear probing and fine- tuning. We also test the transfer of our ImageNet pretrained model (Suppl. Sec. B.1 and B.2).
General Implementation Details. We adopt minimal data augmentation strategy and use the nor- malized pixel value from (He et al., 2021) as our patch regression target. We obtain the reconstruc- tion target by normalizing target pixels using the mean and standard deviation of the patch they belong. Our loss function computes the mean squared error (MSE) between the predicted pixel values and patch-normalized reconstruction target.
Low-data Pretraining. Vision transformers known to be "data hungry" (Dosovitskiy et al., 2021) and require a large dataset and a series of data augmentations to pretrain (Touvron et al., 2021). To experiment in such challenging setting, we evaluate our method on small-scale datasets. We train a "square" and "blob" RandSAC models using 16 + 4 and 11 + 5 hierarchies respectively.
Pretraining ImageNet-1K: ImageNet ILSVVR-2002 (Deng al., 2009) is popular large scale image dataset with 1.28 million images and 1000 categories. We train "square" RandSAC (16 + 4).
Detailed implementation details for all three settings are given in Supplemental Appendix D.
5.1 EVALUATION PROTOCOLS
Linear Probing, This measure is widely used for quantifying the quality of representation learning. It learns linear classifier on top of the frozen feature of pretrained encoder to classify the object- level classification labels. Then performance is evaluated using the val/test set.
End-to-end Fine-tuning. A recent study (Chen al., 2022) shows that linear evaluation favors those methods with a center-bias such as contrastive learning. To complement linear probing, we also include 100-epoch fine-tuning evaluation. fine-tuning, all parameters are optimized for clas- sification. The line-tuning recipe follows the common practice of supervised ViT training.
5.2 RESULTS
8

---

Model
Backbone
Parameter
Linear
Fine-tune
Supervised
DeiT (Touvron al., 2021)
VIT-B
86M
N/A
81.2
Clustering
DINO (Caron et al., 2021)
VIT-B
86M
78.2
82.8
Contrastive Learning
MoCo v3 (Chen et al., 2021b)
ViT-B
86M
76.7
83.2
Masked Image
BEIT (Bao et al., 2022)
ViT-B
86M
N/A
83.2
Modeling
MAB (He et al., 2021)
ViT-B
86M
68.0
83.6
iGPT (Chen et al., 2020a)
iGPT-S
76M
41.9
N/A
Autoregressive
iGPT (Chen et al., 2020a)
iGPT-M
455M
54.5
N/A
Image Modeling
iGTT (Chen et al., 2020a)
iGPT-L
1362M
65.2
N/A
RandSAC RandSAC-Squar AQuuuee (K=9)
ViT-B
86M
72.3
83.7
RandSAC-Squar (K=16->4
ViT-B
68.9
83.9
Table 8: Comparison on ImageNet-1K. Methods except for Autoregressive Image Modeling use image size 224 224. RandSAC uses image size 192 for pre-training and 224 224 for evaluation.
Table 7 shows low-data classifica- tion performance for clustering pre- training (DINO (Caron et al., 2021)), masked image encoding (MAE (He et al., 2021)) and segment toregressive coding (RandSAC). The MAE and DINO pretrained using their official implementations. For 75% masking ratio suggested in their paper. All models- are pretrained for 1600 epochs and evaluated with both 90-epoch linear probing (LIN) and 100-epoch fine-
CIFAR10
CIFAR100
Model
LIN
FT
LIN
FT
Supervised
91.3
64.13
DINO (Caron al., 2021)
89.0
94.4
65.78
76.3
MAE (He et al., 2021)
87.3
95.9
54.0
81.1
LandSAC-Square RandSAC-Blob
92.1
96.7
69.7
81.5
93.9
96.9
67.9
79.6
MAE
Table 7: Low-data pre-training CIFAR 10 and 100.
RandSAC-Square uses 16 + hierarchy while RandSAC- Blob uses 11 + 5.
tuning (FT). Under the low data benchmark, RandSAC outperforms other non-autoregressive al- gorithms and direct supervised training, by a large margin. Both the square and the blob hierarchical versions work well. We postulate that the superior performance of RandSAC comes from random- ized segment prediction pretext task. The autoregressive coding objective that we propose, which is to traverse hierarchy of randomly serialized visual segments, diversifies the small dataset, and serves as sort of data augmentation.
Table 8 shows ImageNet pretraining result. We compare RandSAC with clustering (DINO (Caron et al., 2021)) and contrastive (MoCo v3 (Chen et al., 2021b)) transformer approaches, masked image encoding (BEIT (Bao et al., 2022) & MAE (He al., 2021)), and autoregressive counterpart iGPT (Chen et al., 2020a). We note, that due to limited access to computation, we were only able to AC once, without any parameter tuning. Nevertheless, RandSAC outperforms all predic- tive nonncontraastiv methods) in linear probing, despite using a smaller image size for pretraining (192 224). is also among the best in fine-tuning (on par with MAE and better than the rest).
Contrastive models do tend to perform better in linear probing, but also differ in pre-training, For example, contrastive methods require two global crops of the input image while other methods only process one crop; DINO uses 10 local crops. In addition, linear probing for DINO and iGPT evaluated using the last 4 and transformer blocks, respectively, while MoCo v3. MAE, and AC only evaluate the last block output. A longer feature vector tends to result in better linear probing accuracy (Caron et al., 2021; Chen al., 2020a). Lastly, it worth mentioning that RandSAC can be easily combined with contrastive objectives in the future.
6 CONCLUSION
We present a new self-supervised pre-training strategy we call RandSAC. In doing so, we also study and provide general insights into VIT pre-training (e.g., tokenization, segmentation, and serializa- tion). We found randomized serialization of hierarchical image segments significantly improves autoregressive pre-traie mre-training of ViTs. In addition, propose coder, which facilitates improved performance. We show evidence that the proposed task and model could be the key developing powerful GPT-like model for visual representation learning.
design for the transformer de-

---

ACKNOWLEDGMENTS
This work was funded, in part, by the Vector Institute for AI, Canada CIFAR AI Chair, NSERC CRC, and an NSERC DG and Discovery Accelerator Grants. Resources used in preparing this research were provided, in part, by the Province of Ontario, the Govermment Canada through CIFAR, and companies sponsoring the Vector Institute https: evectorinstitus Additional hardware support was provided by John R. Evans Leaders Fund CFI grant and Com- pute Canada under the Resource Allocation Competition award. Finally, we would like to sincerely thank Muchen Li for valuable feedback and discussions.
ai/ ai/part.ness.
REFERENCES
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zit- nick, and Devi Parikh. Vqa: Visual question answering. In IEEE International Conference Computer Vision (ICCV), pp. 2425-2433, 2015.
Hangbo Bao, Li Dong, and Furu Wei. BEiT: Bert pre-training of image transformers. International Conference Learning Representations (ICLR), 2022.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot leamers. Advances in Neural Information Processing Systems (NeurIPS), 33:1877-1901, 2020.
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. European Conference on Computer Vision (ECCV), 2020.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski. and Armand Joulin. Emerging properties in self-supervised vision transformers. In IEEE/CVF Inter- national Conference on Computer Vision (ICCV), pp. 9650-9660) 2021.
Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and Ilya Sutskever. Generative pretraining from pixels. In International Conference on Machine Learning (ICML), pp. 1691-1703, 2020a.
Ting Chen, Simon Korbblith, Mohammad Norouzi, and Geoffrey E. Hinton. A simple framework for contrastive learning of visual representations. ArXiv, abs/2002.05709, 2020b.
Ting Chen, Calvin Luo, and Lala Li. Intriguing properties of contrastive losses. Advances in Neural Information Processing Systems (NeurIPS), 34, 2021a.
Xiaokang Chen, Mingyu Ding, Xiaodi Wang, Ying Xin, Shentong Mo, Yunhao Wang, Shumin Han, Ping Luo, Gang Zeng, and Jingdong Wang. Context autoencoder for self-supervised representa- tion learning. arXiv preprint arXiv:2202.03026, 2022.
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. 2021 JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 15745-15753 2021.
Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self-supervised vision transformers. IEEE/CVF International Conference on Computer Vision (ICCV), 2021b.
Jean- Caan-Baptiste Cordonnier, Andreas Loukas, and Martin Jaggi. On the relationship between self- attention and convolutional layers. International Conference on Learning Representations (ICLR), 2020.
Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le. Randaugment: Practical automated data augmentation with a reduced search space. In IEFE Conference on Computer Vision and Pattern Recognition Workshops, 2020.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR). pp. 248-255. leee, 2009.
10

---

Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprins 1exivv:810.0440 2018.
C. Doersch, A. Gupta, and A. A. Efros. Unsupervised visual representation learning by context prediction. In IEEE/CVF International Conference on Computer Vision (ICCV), 2015.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image worth 16x 16 words: Transformers for image recognition scale. International Conference on Learning Representations (ICLR), 2021.
Alaaeldin El-Nouby, Gautier Izacard, Hugo Touvron, Ivan Laptev, Hervé Jégou, and Edouard Grave. Are large-scale datasets necessary for self-supervised pre- traininnninn? ArXiv, abs/21 12.407 2021.
Priya Goyal, Piotr Dollár, Ross B. Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia. and Kaiming He. Accurate, large minibatch sgd: Training ima- genet in hour. ArXiv, abs/1706.02677, 2017.
Jean-Bastien Grill, Florian Strub, Florent Altch'e, Corentin Tallec, Pierr H. Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Ávila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, Bilal Piot, Koray Kavukcuoglu, Rémi Munos, and Michal Valko. Bootstrap your own latent: A new approach to self-supervised learning. Conference on Neural Information Processiig Systems (NeurIPS), 2020.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-enn. IEEE International Conference Computer Vision (ICCV), pp. 2961-2966 2017.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross B. Girshick. Momentum contrast for unsupervised visual representation learning. 2020 IEEE/CVF Conference Computer Vision and Puttern Recognition (CVPR), pp. 9726-9735, 2020.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. arXiv preprint arXiv:2111.06377 2021.
Geoffrey E. Hinton. How represent part-whole hierarchies in a neural network. ArXiv, Abs/2102.11627,, 2021.
Tianyu Hua, Wenxiao Wang, Zihui Xue, Yue Wang, Sucheng Ren, and Hang Zhao. On feature decorrelation in self-supervised learning. ArXiv, abs/2105.00470, 2021.
Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q Weinberger. Deep networks with stochastic depth. 2016.
Alex Krizhevsky. Learning multiple layers of features from tiny images. 2009.
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. Albert: lite bert for self-supervised learning of language representations. ArXiv, abs/1909. 1942,, 2020.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. ArXiv, abs/1907. 1692, 2019.
Ze Liu, Yutong Lin, Yue Caol, Han Hul, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In IEEE/CVF International Conference Computer Vision (ICCV), 2021.
Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint /608.00360000380 2016.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. International Conference Learning Representations (ICLR), 2019.

---

Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. VilBERT: Pretraining task-agnostic visi- olinguistic representations for vision-and -ang-agg tasks. In Conference on Neural Information Processing Systems (NeurIPS), 2019.
Bar Mayo, Tamir Hazan, and Ayellet Tal. Visual navigation with spatial attention. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 16898-16907, 2021.
Mehdi Noroozi and Paolo Favaro. Unsupervised learning of visual representations by solving jigsaw puzzles. In European Conference on Computer Vision (ECCV), pp. 69-84, 2016.
Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexei A. Efros. Context encoders: Feature learning by inpainting. In IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2016.
Alec Radford and Karthik Narasimhan. Improving language understanding by generative pre- training. 2018.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models unsupervised multitask learners. 2019.
Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi. You only look once: Unified, real-time object detection. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 779-788, 2016.
Sucheng Ren, Fangyun Wei, Zheng Zhang, and Han Hu. Tinymim: An empirical study of distilling mim pre-trained models. arXiv preprint arXiv:2301.01290 2023.
Martin Rolfs. Microsaccades: small steps long way. Vision research, 49(20))2415-2444 2009.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-Net: Convolutional networks for biomedi- cal image segmentation. In Medical Image Computing and Cosputer-Assisted Intervention (MIC- CAI). pp. 231, 2015.
Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, and Jifeng Dai. VL-BERT: Pre- training of generie visual-linguistic representations. In International Conference Learning Representations (ICLR), 2020.
Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In IEEE Conference Computer Vision and Pattern Recoonittion(CVPR) 2016.
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. In European
Conference on Computer Vision (ECCV), 2020.
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In International Conference on Machine Learning (ICML), pp. 10347-10357, 2021.
Aaron van den Oord, Nal Kalchbrenner, and Koray Kavukcuoglu. Pixel recurrent neural networks. In International Conference on Machine Learning (ICML), 2016.
Aäron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding, ArXiv, abs/1807 1ww/1807.000 .037488 2018.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in Neural Informa- tion Processing Systems (NeurIPS), 30, 2017.
Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre-Antoine Manzagol. Extracting and composing robust features with denoising autoencoders. In International Conference on Machine Learning (ICML), pp. 1096-1103. 2008.
Chen Wei, Haoqi Fan, Saining Xie, Chaoxia Wu, Alan Loddon Yuille, and Christoph Feichtenhofer. Masked feature prediction for self-supervised visual pre-training. ArXiv, abs/21 12.09133, 2021.
12

---

Tete Xiao, Xiaolong Wang, Alexei A. Efros, and Trevor Darrell. What should not be contrastive in contrastive learning. International Conference on Learning Representations (ICLR), 2021.
Zhenda Xie, Zheng Zhang, Yue Cao, Yutong Lin, Jianmin Bao, Zhuliang Yao, Qi Dai, and Han Hu. SimMIM: A simple framework for masked image modeling. /EEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), 2022.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime G. Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. Xlnet: Generalized autoregressive pretraining for language understanding. Conference on Neural Information Processing Systems (NeurIPS), 2019.
Alfred L.. Yarbus. Eye Movements and Vision. Springer, 1967.
Yang You, Igor Gitman, and Boris Ginshurg. Large batch training of convolutional networks. arXiv; Computer Vision and Pattern Recognition, 2017.
Jiahui Yu, Zhe L. Lin, Jimei Yang, Xiaohui Shen, Xin Lu, and Thomas S. Huang. Generative image inpainting with contextual attention. JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5505-5514, 2018.
Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Young Joon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. 1EEE/CVF International Conference Computer Vision (ICCV), 2019.
Jure Zbontar, Li Jing, Ishan Mísra, Yann LeCun, and Stéphane Deny. Barlow twins: Self-supervised learning via redundancy reduction. In International Conference on Machine Learning (ICML), 2021.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. International Conference on Learning Representations (ICLR), 2017.
Richard Zhang, Phillip Isola, and Alexei A. Efros. Colorful image colorization. In European Con- ference on Computer Vision (ECCV), 2016.
13

---

A APPENDIX
A.1 EFFECT OF PATCH SIZE
As we discuss in the main paper (Sec- tion 3.2), the control (mainly reduc- tion) over the number of autoregres- sive steps can be achieved by sim- ply varying the patch size P in the patch-random model. The result of this on CIFARIO are illustrated in Table 9. It can clearly be seen that a different patch size does not lead improved representation for segment- free pat ch-r random prediction task. The segment formation, on the other hand, as we show in the paper, does substan- tially improve the performance.
1
2 x 2
4 x 4
8 x
16 x 16
LIN(1) FT(1)
59.79
69.63
75.53 87.52
75.34 83.10
60.77
79.70
87.18
69.23
Table 9: Patch-random tokenization as a function of P on CIFAR10.
A.2 BLOO SEGMENTS.
We define blob segments irregular elliptical segments defined by sampled Mixture of Gaussians.
To obtain random blobs for a given image, we first sample K Gaussians with means sampled from
(x) (9) ~ U( u1-1.75, 1.75) and standard deviations from [o [ U(0.5, 1), where U/ is (ng Pk
uniform distribution. Then we simply assign each token x, which is at normalized position (24, y1 in the range of 2- 2] (i.e., leftmost top token is at (-2,-2), rightmost bottom token is at (2,2)). The assignment is done as follows:
(1
1)
S(x)) arg arg maxx N 24 Vi
(v)
(0) F
h
S is function that maps tokens to segments. The sampling for both square and blob is only used during segment predictive training and is disabled during evaluation. The computation cost for sam- pling is, comparatively, negligible. Note that beyond the shape, blob segments allow for variability in size squares do not.
A.3 TRAINABLE SKIP CONNECTIONS
We define transformer design, with learnable skip connections, that leverage for main experiments in Section 4.2 of the main paper. Here we provide additional details and evaluation of that design which is illustrated in Figure 5 (right).
A transformer with Lene encoder layers and Ldec decoder layers processes input X into Lene hidden (hi,-.. ). In traditional masked transformer, decoder memory is set to Z' = H ene for each layer I of the decoder. Instead, we introduce a linear attention layer that allows each decoder layer attend over encoding hidden representations. In other words, we learn linear layer with parameters W X such that: Z' Ek=1 combined memory cells are conditioned on, and different, for each individual decoder layer.
representations Hi
L.ene Wi.
ene. Note, the linearly
ENCODER
ENCODER
DECCDER
ENCODER
DECODER
1
4
1
ENCODER
ENCODER
DECODER
ENCODER
DECODER
t
1 ENCODER
ENCODER
ENCOOER
Figure 5: Candidate architectures for autoregressive segment prediction: Left: Two-stream Transformer (Yang al., 2019). Middle: Masked Transformer. Right: proposed Masked Trans- former with Trainable Skip Connections.
14

---

CIFARIO LIN (1)
ImageNet100
Enc
Dec
(1)
Enc
Dec
LIN (1)
FT (f)
Two-stream
7M
0M
84.4
91.5
30M
OM
62.4
84.6
Transformer
5M
2M
88.0
93.6
21M
9M
65.8
85.9
Transformer-skip
5M
2M
89.5
94.4
21M
9M
70.7
87.3
Table 10: Performance of Architectures for RandSAC. See text for details.
To evaluate the effectiveness of the proposed Masked Transformer and trainable skip connection layer for segment prediction, compare three architectures:
Two-stream Transformer. This design was proposed in (Yang et al., 2019) for permutation-base language modeling. enables randomized target predictions by leveraging a two-stream attention layer: the content stream encodes the full contextual information, and the query stream, which only has access the previous content, is designed to make current predictions. We apply this architecture for our segment prediction by setting the content mask with our "source mask" and query mask with our "memory mask". Model weights for both content stream and query stream are shared (see Figure 5 (left) for illustration of design).
Masked Transformer. For masked transformer we utilize architecture described in Section 4.1 and illustrated in Figure 4; also in Figure 5 (middle). Compared with the Two-stream Transformer above, this design enables communication among jointly predicted tokens within a segment. Also, compared with Two-stream Transformer, weights for encoding and decoding the segment content are decoupled in the Masked Transformer.
Masked Transformer with Trainable Skip Connections. A Masked Transformer only decodes based on the (last layer) encoder output. A trainable skip connection layer we introduce dynam- ically allocates memory assignments between intermediate layers of transformer-encoder-deeom (see Figure (right)). As can be seen from the results in Table 10, this variant does outperform the two competitors both CIFARIO and ImageNet100 datasets. Compared with Masked Transformer, the additional computation cost introduced by trainable linear layer is almost negligible (Table 11).
Two-stream
Transformer
Transformer-skip
VIT-T
499.60 M
483.82 M
484.26 M
ViT-S
10.94 G
6.73 G
6.77 G
ViT-B
35.74 G
21.25 G
21.49 G
Table 11: FLOPs of Three Candidate Architectures.
B EXPERIMENTS
B.1 SEMANTIC SEGMENTATION ON ADE20K.
We take our pretrained backbone as initialization and end-to-end fine-tune with UpperNet frame- work ADE20K to evaluate the performance of our pretrained model on downstream task. semantic segmentation. We follow the same setting of BeiT' (Bao et al., 2022). We compare our pre-training with DeiT' (Touvron et al., 2021), MoCo (Chen al., 2021b), DINO (Caron et al., 2021), BeiT (Bao et al., 2022), MAE (He et al., 2021) in Table 12. Our pre-training outperform DeiT, MoCo, DINO, BeiT, MAE by 1.5, 1.3, 1.3, 2.0 and 0.4, respectively.
B.2 OBJECT DETECTION COCO
We take our pretrained model as initialization and finetune with Mask RCNN on COCO. To adapt the the four stages designs with strides 4, 8, 16, 32 FPN backbone in Mask RCNN, we evenly divide all 12 Transformer blocks into 4 subsets and apply convolutions to upsample or downsam- ple the intermediate feature maps for producing same scales as the requirement of FPN backbone. The results MoCo (Chen et al., 2021b), DINO (Caron et al., 2021), 1), BeiT (Bao et al., 2022), and MAE (He et al., 2021) by 3.0, 3.0, 4.1, 1.1, 0.6 Apidoz and 2.1, 2.3, 3.5, 0.6, 0.1 Apmask
reported in Table 13. Our pre-training outperform DeiT (Touvron al., 2021),
15

---

Method
Crops
Super.
Self-super.
mloU
DeiT (Touvron et al., 2021)
47.0
MoCo v3 (Chen al., 2021b)
2
/
47.2
DINO (Caron et al., 2021)
2+10
47.2
BEIT (Bao al., 2022)
V
46.5
MAE
X
48.1
RandSAC-Squaal (K=9)
48.3
RandSAC-Squen (K=16->4)
X
/
48.5
Table 12: Semantic Segmentation ADE20K
Method
Pre- Fre-Epoohs
Apbbor
Apmesk
DeiT (Touvron et al., 2021)
300
47.9
42.9
MoCo-v3 (Chen et al., 2021b)
300
47.9
42.7
DINO (Caron et al., 2021)
300
46.8
41.5
BETT (Bao et al., 2022)
800
49.8
44.4
MAE (He al., 2021)
1600
50.3
44.9
AC AC-Squree (K=16-4)
1600
50.9
45.0
Table 13: Object Detection COCO
B.3 VISUALIZATION OF RECONSTRUCTION ON IMAGENET- 1K VALIDATION SET
We visualize for both RandSAC-Blot and RandSAC-Square reconstruction results below.
B.4 IMPLEMENTATION DETAILS
We describe implementation details omitted from the main paper due space limitations here.
Implementation Details. We adopt minimal data augmentation strategy following (He et al., 2021):
resize cropping with scale range of (0.2, 1.0) and aspect ratio sampled within range [ $). followed by 50% chance random horizontal flipping. We do not use color jittering, path dropping, or gradient clip in pretraining. We use Adam W as optimizer and pretrain RandSAC for 1600 epochs. We use linear Ir scaling rule (Goyal et al., 2017) that scales the base Jr by batchsize 256.. The Ir scheduled to warm-up from base Ir, then decayed following a cosine-decay rule (Loshchilov & Hutter, 2016). For both benchmarks, 2021) as our patch regression target. Our loss function computes the mean squared error (MSE) between the patch-normalized reconstruction and original image pixels.
the normalized pixel loss introduced from (He al.,
B.5 EVALUATION PROTOCOLS
Linear Probing. Note that the dimension of the feature that the classifier is trained may influence the eventual accuracy readout (Caron et al., 2021). A longer feature vector likely to produce a better linear result. Prior works such (Caron al., 2021) concatenate the feature vectors from the last ViT blocks and (Chen et al., 2020a) use feature vectors up to 15360 dimensions for evaluation. We, however, use only the last encoder averaged feature output following (He et al., 2021; Chen et al., 2021b) (i.e., 384 dimensions for ViT-S and 768 dimensions for ViT-B). The linear classifier is trained for 90 epochs.
C DETAILS FOR SECTION 3
C.I PRE-TRAINING SETTINGS FOR CIFARIO 10 AND IMAGENET100
The following is the experiment configurations for CIFARIO and ImageNet100 from Section 3 of the main paper, including Table and 10 in Appendix. Details for end-to-end fine-tuning and linear
16

---

probing the same with ImageNet-1K, IK. For Table 10, both CIFARIO and ImageNet100 experi- ments are trained using a "Blob" RandSAC model using hierarchy 11 -5.
C.I.1 CIFARIO EXPERIMENTS.
The default setting illustrated in Table 14. We pre-train ViT-Tiny encoder on CIFARIO. The ViT- Tiny has 12 layers. Each layer has 192 dimensions and 3 self-attention heads. We chose patch size x 4 and split the 32 32 images into 8 x 8 tokens. For segment decoding, we use transformer decoder layers following the same configuration for the encoder.
config optimizer base Ir weight decay ,1,
value
AdamW (Loshchilov & Hutter, 2019)
0.001
0.05
0.9, 0.999 (Carion et al., 2020)
batch size learning rate schedule warmup epochs training epochs augmentation
512
cosine decay (Loshchilov & Hutter, 2016)
10
800 (Table 1-6, 9) 1600 (Table 10)
RandomResizedCrop False (Table 1-6, 9) True (He et al., 2021) (Table 10)
vorm_pixel_lor loss
Table 14: CIFAR10 Pre-training setting.
C.1.2 IMAGENETION 100 EXPERIMENTS.
Experiments that involve ImageNet100 are Table 2, 4, and 10. We pre-train ViT-Small encoder on ImageNet100 (Tian et al., 2020). The ViT-Small backbone has 12 layers. Each layer has 384 dimensions and self-attention heads. We chose patch size x 16 following (Dosovitskiy et al., 2021) and split the 224 x 224 images into 14 x 14 tokens. For segment decoding, we use 4 layer transformer decoder and double the attention heads while keeping all other configurations the same as the ViT-Small encoder.
config optimizer base_Ir weight decay
value
AdamW (Loshchilov & Hutter, 2019)
1.5e-4 (He et al., 2021)
0.05
31.32 batch size
0.9, 0.95 (Chen et al., 2020a)
4096
learning rate schedule warmup epochs training epochs augmentation
cosine decay (Loshchilov & Hutter, 2016)
800
RandomResszedCrop
norm-pixel_les llos
True
Table 15: ImageNet100 Pre-training setting.
D DETAILS FOR SECTION 4
D.I LOW-DATA PRE-TRAINING SETTING
We pre-train ViT-Small on CIFAR10 and CIFAR100 (Krizhevsky, 2009). Both datasets small- scale image datasets containing 60000 32 32 images that belong to 10 and 100 categories, respec- tively. The Small has 12 layers. Each layer has 384 dimensions and 6 self-attention heads. We chose patch size x 4 and split the 32 x 32 images into 8 x 8 tokens. For segment decoding, we transformer decoder layer. The attention-head and feature dimensions of the decoder the
17

---

the encoder. We also set the decoder for MAE (He al., 2021) have the same depth, attention head, and dimension as ours.
config optimizer base Ir weight decay
value
AdamW (Loshchilov & Hutter, 2019)
0.001
0.05
31,82
0.9, 0.999 (Carion et al., 2020)
batch size
512
learning rate schedule warmup epochs training epochs augmentation
cosine decay (Loshchilov & Hutter, 2016)
10
1600
RandomResizzeCCnpp
Norm-pixel_loss
True (He et al., 2021)
Table 16: Low Data Pre-training setting.
D.2 IMAGENET-IK PRE-TRAINING SETTING
We resize the images 192 192 during pretraining and set patch size P be 16. We pretrain
Square-RandSAC with hierarchy 16 -44 using ViT-Base (Dosovitskiy et al., 2021) on ImageNet- 1K following (Bao al., 2022: He al., 2021). ViT-Base model has 12 blocks, with each block having dimension 768 and 12 heads. We chose an 8 layer decoder. The attention-head and feature dimensions of decoder are the same as encoder.
config optimizer base Ir weight decay
value
AdamW (Loshchilov Hutter, 2019)
1.5e-4 (He et al., 2021)
0.05
$1.82 batch size learning rate schedule warmup epochs training epochs augmentation
0.9, 0.95 (Chen et al., 2020a)
4096
cosine decay (Loshchilov & Hutter, 2016)
40
1600
Raniommesiizedrrpp
norm_pixel_los _loss
True
Table 17: ImageNet-IK pre-training setting.
D.3 EVALUATION CONFIGURATIONS
Different from VIT (Dosovitskiy et al., 2021), where additional class token is required for classifi- cation, we directly use the averaged pooled feature out of the encoder for both fine-tuning and linear probing. The hyper-parameters for both end-to-end finetuning and linear probing from Table 18 and Table 19 are used for all experiments of this paper.
E IMPLEMENTATION DETAILS OF SEMANTIC SEGMENTATION
We end-to-end fine-tune our pre-trained VIT encoder with UpperNet framework ADE20k to evaluate the performance on downstream task, semantic segmentation. We follow the same setting of BeiT (Bao al., 2022). We take AdamW as the optimizer and set the batch size to 16, the layer- wise decay rate to 0.65, the input resolution to 512 x 512, fine-tuning iterations are set to 160K steps. During evaluation, do not take multi-scale testing strategy in our experiment.
18

---

config
value
optimizer
Adam
base_Ir
5e-4
weight decay
0,05
$1,82
0.9, 0.999 (Chen et al., 2020a)
layer-wise Ir decay batch size learning rate schedule warmup epochs
0.65
1024
cosine decay (Loshchilov & Hutter, 2016)
5
training epochs
100
augmentation
RandAug (9, 0.5) (Cubuk al., 2020)
label smoothing (Szegedy et al., 2016)
0.1
mixup (Zhang al., 2017)
0.8
cutmix (Yun et al., 2019)
1.0
drop path (Huang al., 2016)
0.1
Table 18: End-to-end fine-tuning setting.
config optimizer
value
LARS (You et al., 2017)
base_Ir Ir weight decay momentum
0.1
batch size learning rate schedule
16384 cosine decay
warmup epochs
10
training epochs augmentation
RandomResiieeCCrpp
Table 19: Linear probing setting.
F CODE AND REPRODUCIBILITY
We include an implementation of RandSAC- AC-Suuree model using PyTorch. We will release the com- plete training/evaluation code and all pre-trained models upon acceptance of the paper.
import torch
import torch.nn as nn import corch.nn. functional as import Rearrange from cinops import rearrange
from
from torch import Tensor from typing import Optional
class Transformer_stt (nn. Transffrmerr: int = 4, **kwargs):
def init init_(seeE,, num numcene eaceeer_llaerss: int 6, num decoder layers:
11
"""Transformer with learnable skip connects between encoder and
decoder. o
12
super () . init htt.tt/wmmpooeee...eeeeeeoooooee
13
14
norm_first-True, **wargg))
15
self.skip self.skip_conmectiice = nn. Linerr) nun_encoder_layers mmm_decoddr_llayerg
18
def (sewa, sre: Tensor, tgt: Tensor, sre_mask: Optional [Tensor
None, tgt_mask: Optional (Tensor) = None, memory_mask: Opticnal [Tessor] None) -> Tensor:
19
Forward encoder layers
19

---

memory
23
for layer self. encoder layers:
24
src layer (src, 2ro_mask-sro_mask memory. (src)
25
(src)
memory. sel self. encoder. norm (torch, stack (memory)
28
29
Dynamic memory assignment
30
memory self wwtptiskpp_comoee connection(
memory.Elatten (1) transpose (0, trrasspose (0, 1) view (--1, *meoory [0] shape))
32
33
34
Forward decoder layers for i, layer in enumerate (self. decoderecrrr.... layers):
tgt eayer(tgt, memorylil,
37
tgt jtt_mask=tgt_mask memory_masee sunory_mak--remoryymoo _aask
38
39
return self. aecoodr..normtee..
class RandSAC (nn Module)
42
def init (self, d_model, image_channel-3, image_size-192 patch_size-1l 16 M-4, attransformer_kwargee
43
super initt_ **"
45
RandSAC implementation with square segments and flat serialization (no hierarchy).
46
grid_size siae // patch_size
patch_ding patch_ patch_ size image_channel
49
50
self.M M
51
sclf.patchify Rearrange
'n (h pl) (w p2) -> h (pl o)', pl-patch_size, p2=
patea_size)
53
self.in ino_jjj Lineir (patch_ding d_model)
54
self. transformer Transformer_skipl d_model= vxtransformer_kwarrgs)
37
58
self,out proj Liner (d model,,, patch dim)
self.pos = Pa (torch. zeroos zeros (1, grid_size
d model)
60
terch.nn. init
(self. norta
std=.02)
61
62
self, rer.r.erister_bbfeeee.ee
'mask', torch. repeat__intereevee torch. repeat_interlesst
nn mn.Transformer gww.earr.e........e............. _errerate_seguurrerr_ee_eeee equare_subsequent
66
sz=grid // M+*2
67
2 dim-0
70
repeats-M dim-l
71
74
def scrialize (self, patches):
75
"""Flat scrialization
76
d1, d2 patches. shape(-1), self pos. shape [-1]
tckens torch. [patches, self. wo..ppo...... repeat (patches. srtcches.shhpe..e. [0], 1, 1)], din
78
repeat
79
seq = rearrange
tokens, 'n (h ml) (w m2) d -> (h w) ml m2 d', ml=self.M, m2
=self
20

---

G VISUALIZATION OF TOKENIZATION AND SERIALIZATION noise torch. rand (*seq. shape (
device=seq device)
82
ids_shuffle argsett (noise, dim=1)
83
seq torch gather (seq, dim=1, fedox-ide_shuffle... view
84
sseq. 1:2), 1, 1, 1) as (seq))
seq. flattnn (1, 3) transspoee (0, 1). spllit(([d),
d2), din--1)
87
88
def (self, img, label=Ncne):
"""Forward RandSAC***
patches self patchiattiity (img)
patches, pos self.seri al aze (patches)
92
93
seg_size self.M+*2
embedings self. in proj (patches)
96
dec_out self. transforrrerrre (src= (embedinng pos) [: [:-seg_size] tgt =oos [scg_aize:),
97
src_mask-self.man mask, tgt_mask=self mma. mask
memory_mask=selff mask)
99
pixel_recom self.out_pren (dec_out)
100
loss = F F.se loss soixel_recon patches seg siz_size:l
return loss
We visualize different tokenization and serialization schemes, discussed in Section 3 of the main paper, in the video file included as part of the supplemental materials.
21

---

Ground Truth
Blob
Square
Ground Truth
Blob
Square
Ground Truth
Blob
Square
Figure Visualization of image reconstruction from "Blob" and "Square" RandSAC
22


Reviews:
[{"review_id": "xNs1cKvM_g", "reviewer": "Reviewer_pLid", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper focuses on autoregressive visual pre-training. The authors propose to spatially group tokens into a larger-resolution unit called segment, and then arrange them into a random order. After that, the authors sequentially predict the segments using transformer architecture with skip connection. The cifar dataset is used as the main benchmark to show the effectiveness of the proposed components.", "summary_of_the_review": "+ Currently, the experimental results do not strongly prove the potential of the proposed framework in visual pre-training. The reviewer encourages the authors to conduct experiments with ImageNet and larger model since the scale effect is important for pre-training.\n+ The authors should try to finetune their model on other downstream tasks beyond the same-dataset classification, in order to verify the gap between the proposed autoregressive pre-training and downstream transfer.\n+ The authors should improve the writing and show how the proposed methods are strongly aligned with the autoregressive mechanism.\n", "strength_and_weaknesses": "Issues about the autoregressive mechanism.\n+ In NLP, the autoregressive pre-training paradigm might have more benefit for generative downstream tasks. The reviewer suggests the authors to elaborate more on this point and compare with the autoencoding counterpart. The reviewer believes that such discussion will make the paper standing out.\n+ Currently, the authors simply demonstrate the effectiveness of this mechanism by showing the performance on downstream classification. The reviewer want to have more comprehensive impressions of the pretraining-finetuning gap of this mechanism. Could the authors conduct more downstream tasks such as detection and segmentation?\n+ Since this paper focuses on the pre-training, the reviewer believes that using cifar dataset as the main example through the paper is somehow not convincing. As it is widely recognized that the scaling effect is very common in terms of the pre-training data, parameters, etc. As shown in the paper, the benefit of the proposed components become unsignificant when the data scale is increasing. To this end, the reviewer encourages the authors to use ImageNet as the main example and try to verify on models with larger capacities, such as the standard variant of MAE (i.e., ViT-Large).\n***\nQuestions about the framework.\n+ The reviewer finds that the decoder is heavier than that of MAE, since the dimensions of its hidden embeddings are enlarged to match with the encoder. Could the authors clarify how much the extra overhead this framework brings when compared with MAE?\n+ Could the authors explain why the proposed methods are strongly aligned with the autoregressive modeling? The reviewer finds that the some of them, such as grouping the tokens as a segment or the skip connection in the transformer, can be applied to autoencoding modeling as well. The reviewer suggests the authors to show their strong connections with the autoregressive mechanism.\n***\nOther issues needed to be addressed.\n+ In section 3, why the authors divide the tokenization strategy of BEiT and MAE into two categories. In fact, they share the same implementation to patchify images.\n+ For table 7, there are also many contrastive learning based methods used on cifar. The authors can include them for comparison.\n+ Current MAE can be regarded as predicting all the masked tokens sequentially at only one time. One the other hand, MAE can also be adjusted that reconstructing a partial ratio of masked tokens at every time sequentially. From the finetuning performance, improvement over MAE cannot be seen. Could the authors show the superiority of the proposed method against MAE?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "aahkQhyCh2", "reviewer": "Reviewer_qvKL", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents an interesting method continuing the previous auto-regressive self-supervised learning schema. Unlike the existing iGPT which operates on pixel-leve in a fixed order, RandSAC proposes to regress the image tokens in sequential (across grouped image segments) and parallel fashions (within segment). Surprisingly, with this simple method, the exhibited empirical results are optimistic and much efficient compared with iGPT. ", "summary_of_the_review": "The presented RandSAC builds upon the previous auto-regressive self-supervised visual representation fashion (i.e., iGPT). Compared with existing work, the proposed grouping and regressive fashion is simple but effective. The proposed hierachical grouping strategy brings obvious goodness: 1. allows various length learning on visual segments. 2. It largel improves the efficiency of pre-training. \n", "strength_and_weaknesses": "Strength:\n1. Great paper writing with clear and easy understanding motivation.\n2. Good empirical results and comprehensive ablations.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "Gqn93-CS6wJ", "reviewer": "Reviewer_FF3A", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper delves deep into ordering and grouping methods for causal image modeling. Moving on from iGPT, which used quantized RGB tokens for its causal prediction, this paper lets the models predict the pixel values directly as MAE does. When it comes to ordering, the paper showed that random ordering could yield much better performance compared to conventional raster ordering. And for the grouping (segmentation) method, two approaches were square segments and blob segments, and both performed better than no segment baseline. Further, the paper proposed a learnable skip connection module connecting the transformer encoder and decoder layers.", "summary_of_the_review": "It would be nice if some of my questions (see C2-C4) were answered, but I want to give this paper a high recommendation because its strengths far outweigh its weaknesses.", "strength_and_weaknesses": "- (C1) There have not been many papers studying ordering or grouping methods in autoregressive methodologies, I would like to give a high score to find a way to do it well.\n- (C2) I believe DINO [1] is not a contrastive learning method but instead bootstrapping or self-distillation-like method (e.g., BYOL [2]). We can find another stream of works, such as SwAV [3] and DeepCluster [4,5], which are not contrastive learning, masked image modeling, or bootstrapping (these methods are better classified as a clustering method). Thus, coarsely dividing SSL methods into contrastive and predictive categories is not appropriate.\n- (C3) CIFAR10 and CIFAR100 experiments in table 7 are rather low-resolution pre-training than low-data pre-training. Although it is a worthy experiment, I think the authors can do an eye-to-eye low-data pre-training by limiting the number of examples for the ImageNet-1K dataset.\n- (C4) I am confused about whether or not the learnable skip connection module is used for the experiments in section 5. Plus, is it okay for the learned matrix W to be trained without any constraints (e.g., \\sum_{k=1}^{L_{enc}} W_{l, k} = 1)? If so, will the constrained version worsen the performance?\n- (C5)\n  - Typo: (Section 5) Suppl. Sec. D.3 -> B.1.\n  - Typo: (Section 5) Supplemental Appendix B → D\n\n[1] Caron, Mathilde, et al. \"Emerging properties in self-supervised vision transformers.\" *Proceedings of the IEEE/CVF International Conference on Computer Vision*. 2021.\n[2] Grill, Jean-Bastien, et al. \"Bootstrap your own latent-a new approach to self-supervised learning.\" *Advances in neural information processing systems* 33 (2020): 21271-21284.\n[3] Caron, Mathilde, et al. \"Unsupervised learning of visual features by contrasting cluster assignments.\" *Advances in Neural Information Processing Systems* 33 (2020): 9912-9924.\n[4] Caron, Mathilde, et al. \"Unsupervised pre-training of image features on non-curated data.\" *Proceedings of the IEEE/CVF International Conference on Computer Vision*. 2019.\n[5] Caron, Mathilde, et al. \"Deep clustering for unsupervised learning of visual features.\" *Proceedings of the European conference on computer vision (ECCV)*. 2018.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster