SASSL: Enhancing Self-Supervised Learning via Neural Style Transfer
Authors: Renan A. Rojas-Gomez
Karan Singhal
Ali Etemad
Alex Bijamov
Warren Richard Morningstar
Philip Andrew Mansfield
Venue: Accepted by TMLR ()

Abstract: Existing data augmentation in self-supervised learning, while diverse, fails to preserve the inherent structure of natural images. This results in distorted augmented samples with compromised semantic information, ultimately impacting downstream performance. To overcome this limitation, we propose SASSL: Style Augmentations for Self Supervised Learning, a novel data augmentation technique based on Neural Style Transfer. SASSL decouples semantic and stylistic attributes in images and applies transformations exclusively to their style while preserving content, generating diverse samples that better retain semantic information. SASSL boosts top-1 image classification accuracy on ImageNet by up to 2 percentage points compared to established self-supervised methods like MoCo, SimCLR, and BYOL, while achieving superior transfer learning performance across various datasets. Because SASSL can be performed asynchronously as part of the data augmentation pipeline, these performance impacts can be obtained with no change in pretraining throughput.

Published in Transactions on Machine Learning Research (11/2024)
SASSL: Enhancing Self-Supervised Learning via Neural Style Transfer
Renan A. Rojas-Gomee
renanar2@rllinois.ee
University of Mlinois at Tebana-Champonn
Karan Singhal
kemansinghhl@@ggogll..cmm
Google Research
Ali Etemad Queen's University, Canada
ali. adi.ctemmad@luueenn...c
Alex Bijamov Google DeepMind
abijamov@gongle.com
Warren R. Morningstar Google Deep
wmorning@ggoggle.com
Philip Andrew Mansfield Google DeepMind
memes cernnes@googgecco
"Work done during an internship at Google.
Reviewed on OpenReview: https: // openreview. net/ forum? id= NrhXtkPYsk
Abstract
Existing data augmentation self-supervised learning, while diverse, fails to preserve the inherent structure of natural images. This results in distorted augmented samples with com- promised semantic information, ultimately impacting downstream performance. To over- come this limitation, we propose SASSL: Style Augmenlations for Self Supervised Learning, novel data augmentation technique based on Neural Style Transfer. SASSL decouples semantic and stylistic attributes in images and applies transformations exclusively to their style while preserving content, generating diverse samples that better retain semantic in- formation. SASSL boosts top-1 image classification accuracy on ImageNet by up percentage points compared established self-supervised methods like MoCo, SimCLR, and BYOL, while achieving superior transfer learning performance across various datasets. Because SASSL can be performed asynchronously part of the data augmentation pipeline, these performance impacts can be obtained with no change in pretraining throughput.
1 Introduction
Data labelling is challenging and expensive process, which often serves barrier to build machine learning models to solve real-world problems. Sclf-supervised learning (SSL) is an emerging machine learning paradigm that helps to alleviate the challenges of data labelling, by using large corpora of unlabeled data to pretrain models to learn robust and general representations. These representations can efficiently transferred to downstream tasks, resulting performant models which can be constructed without access large pools labeled data. SSL methods have shown promising results in recent years, matching and in cases exceeding the performance of bespoke supervised models with small amounts of labelled data.

---

Published in Transactions on Machine Learning Research (11/2024)
Fassk
Coatent %
Stylized Sey
A
Btatsoous Crop
Solertice
t
c,11
f
CIN:
Alaumenn
Eshr
Stylization Network (7)
Say It's aiaasor
Soyle
Color Jitser
Festure aligumectt:
Rcss
F
Z
Feature bleuding:
Roudon
Gragy
(1 o1s, 1 oz,, Mi Oman
How Flep
Image interpolation:
Extractor
8ik, + Piin fl Wifhar Braed.
(a) Data augmentation pipeline.
(b) Style Transfer preprocessing.
We propose SASST, novel augmentation technique that leverages Style Transfer to create pretraining views that are semantically awarc, focusing solcly on modifying the image's appearance while preserving its content. SASSL combines the image's content with the texture of an external reference style, generating augmented views that better retain the image's semantic meaning. By incorporating Style Transfer into traditional SSL, augmentation pipelines and controlling the stylization strength through gradual blending of style features and pixel values, SASSL promotes stronger representations compared to well-established SSL methods.
Figure 1: Towards diverse SSL data augmentation via Neural Style Transfer.
Given the lack of labels, SSL relies on pretext lasks, i.e., predefined lasks where pseudo-labels can be generated. These include contrastive learning (Chen et al., 2020a; He et al., 2020), clustering (Caron al,, 2021; 2020; Assran ct al., 2022), and generative modcling (He al., 2022; Devlin et al., 2018). Many pretext lasks involve training the model to distinguish between different views of the same input and inputs corresponding to different samples. For these tasks, the way input data is augmented is crucial to learn useful invariances and extract robust representations (Chen et al., 2020a). While state-of-the-art augmentations incorporate a wide range of color, spectral and spatial transformations, they often disrogard the natural structure of an image. a result, SSL pretraining methods may generate augmented samples with degraded semantic information, and may be less able capture diverse visual attributes.
To tackle this challenge, propose Style Augmentations for Self Supervised Learning (SASSL), novel SSL data augmentation technique based on Neural Style Transfer to generate semantically consistent augmented samples. In contrast to augmentation techniques operating specific formats (c.g. pixel spectral domain), SASSL disentangles an image into perceptual (style) and semantic (content) representations that are learned from data. Applying transformations only to the style of an image while preserving its content, we can generate images with diverse appearance that retain the original semantic properties.
Our contributions:
We propose SASSL, novel data augmentation technique based on Style Transfer that naturally preserves semantic properties while diversifying style (Section 4).
We empirically show improved downstream performance on ImageNet Deng et al. (2009) by incorporating SASSL in methods such as MoCo, BYOL and SimCLR without hyperparameter tuning (Sections 5.1, 5.4). We show SASSL learns stronger representations by measuring their transfer learning capabilities on various datasets. Our method boosts linear probing performance by up 10% and fine-tuning by up 6% on out-of-distribution tasks (Section 5.2).
2 Related Work
2.1 Data Augmentation in SSL
Typical data augmentation methods applied to vision tasks include image cropping and resizing, flipping, rotation, color augmentation, noise addition, and solarization. Examples of methods using these are MoCo (He et al., 2020), SimCLR (Chen et al., 2020a), BYOL (Grill et al., 2020), and SimSiam (Chen He, 2021), among others. Other work (Caron al., 2020) shows how generating additional augmentations using this strategy improve performance relative to two view approaches, though this strategy decreases throughput
2

---

Published in Transactions on Machine Learning Research (11/2024)
and requires additional compute. Other rescarch explore how to select augmentation hyperparameters to learn more robust and general features (Wagner et al., 2022; Reed et al., 2021; Tian et al., 2020). In contrast, SASSL proposes new preprocessing technique that can be incorporated in existing augmentation pipelines, boosting performance without additional hyperparameter tuning or changes in pretraining throughput.
Previous SSL work has explored semantic-aware augmentation approaches. Purushwalkam & Gupta (2020) leverage natural video transformations occuring in videos as an alternative to learning from object-centric datasets. Lee et al. (2021) introduces auxiliary loss to capture the difference between augmented views, leading to better performance on tasks where semantic information is lost due to aggressive augmentation. Bai et al. (2022) propose an alternative augmentation pipeline to prevent loss of semantics by gradually increasing the strength of augmentations. While these methods modify the pretraining loss and require keeping track of augmentation hyperparameters, SASSL integrates seamlessly into existing pipelines without additional loss terms or auxiliary data. Our method complements default data augmentation pipelines with a content-preeerving transformation to obtain stronger image representations.
2.2 Neural Style Transfer
Recent image generation and Style Transfer algorithms (Liu et al., 2021; Heitz et al., 2021; Jing et al., 2020; Yoo et al., 2019; Risser et al., 2017; Gatys et al., 2017; Chen et al., 2021a) use CNNs to measure the texture similarity between two images terms of the distance between their neural activations. Feature maps of a pretrained classifier such as VGG-19 (Simonyan & Zisserman, 2014) are extracted and low-order moments of their distribution (Kessy et al., 2018; Huang & Belongic, 2017; Sheng et al., 2018) are used as texture descriptors. By matching such feature statistics, these techniques have shown promising results transferring texture between arbitrary images, improving over classic texture synthesis approaches (Portilla & Simoncelli, 2000; Zhu et al., 2000; Heeger & Bergen, 1995).
A large body of work focuses on artistic applications, reproducing an artwork style over a scene of interest. These methods adopt cither (i) an iterative optimization approach (Risser et al., 2017; Gatys et al., 2016; 2017; Li et al., 2017a), where an initial guess is gradually transformed to depict a style of interest or (ii) autoencoding approach (Liu et al., 2021; Li et al., 2017b; Chen et al., 2021a; Wang et al., 2020), where one or more CNN image generators are trained to impose a target texture in a single forward pass. While selecting an approach implics a tradc-off between synthesis quality and computational cost, in both cases the generaled stylization shows an unnatural appearance, i.e., it often lacks the qualities of a real-world scene.
In the context of data augmentation for supervised learning, Geirhos et al. (2018) and Zheng et al. (2019) addressed texture bias and generated training samples via pre-stylized datasets or stylizing using small collection of style images. Hong al. (2021) explored Style Transfer to improve robustness against adversarial attacks. Jackson et al. (2019) incorporated Style Transfer as transformation in the augmentation pipeline. While their approach of randomly mixing content and style representations yicld promising results, it neglects potential distortions introduced by the Style Transfer network bottleneck. SASSL, in contrast, integrates Style Transfer in a self-supervised setting. Our approach generates diverse augmentations via either pre- computed style representations from external datasets or in-batch stylization with training samples as style references. Importantly, SASSL preserves semantic information through pixel interpolation and feature blending, mitigating the loss of details inherent in Style Transfer networks.
3 Preliminaries
3.1 Superveee Learning
Traditional SSL methods learn compressed representations by maximizing the agreement between differently augmented views of the same data example in a latent space. They do this following the template originally proposed by SimCLR (Chen et al., 2020a). In this setup the input is split into multiple views using data augmentations, encoded into a reprosentation, and then further projected into an embodding over which the loss is computed. There are many potential augmentations that can be used, including (but not limited to) random cropping, flipping, color jitter, blurring, and solarization. By maximizing the similarity of
3

---

Published in Transactions on Machine Learning Research (11/2024)
augmented training samples, the network learns to create robust representations that separate meaningful semantic content from simple distortions that could occur in the real world, and which should not affect the semantic content of an image.
Given a batch of input images (i 2N augmented samples are generated by applying distinct trans- formations to each image. These transformations correspond to the same data angmentation pipeline. Let R correspond to all possible augmentations. Then, positive pairs correspond to augmented views of the same input sample, and negative pairs correspond to views coming from different imput images. Based on this, the 2N augmented samples { can be organized so that indices | 2k and 2k correspond to views of the k-th input sample
22k-1 = r(ie), izk = f(ix); 6r R
(1)
Once augmented views are obtained, a representation computed using an image encoder (typically a CNN model). The representations are then fed to projection head which further compresses them into lower- dimensional manifold where different views of the same image are close together and those from different images are far apart. Let I and g be the encoder (e.g. a ResNet-50 backbone) and projection head (e.g. an MLP layer), respectively. Then, embeddings are obtained for each augmented sample as = go h(ū)).
SimCLR uses the normalized temperature-scaled cross entropy loss (NT-Xent) to learn how to identify positive pairs of augmented samples. First, the cosine similarity every pair of embeddings is computed
(Zm, Zn,)
mmn
pm k
The model is then trained using a contrastive loss by comparing the embeddings of positives, forcing them to be similar to each other. Since the loss is normalized, it naturally forces the representations of views from two different images (negatives) to be distant from each other.
exp(sm.n/ //T)
L
Σ
[[2k 1=1
2k) + 8(2k, 2k 1)], l(m, n) -log
(3)
2N
mim exp( (op(sm///T)
1=1
where T E IR++ is the temperature factor and 1 the indicator function. While SimCT. is a simple framework, it pushed the state-of-the-are significantly on a wide range of downstream tasks including image classification, object detection, and semantic segmentation.
Follow up works to SimCLR such MoCo (Chen et al., 2020b; 2021b), BYOL (Grill et al., 2020) and SimSiam (Chen & He, 2021), among others (Caron et al., 2020; 2021; Assran et al., 2022; Zbontar et al., 2021; Bardes et al., 2021), have largely maintained this template, but have proposed modifications to this setup (e.g. new losses, architectures, or augmentation strategies) which attempt to further improve the downstream task performance.
3.2 Neural Style Transfer
Style Transfer techniques combine the semantics (content) of an image with the visual characteristics (style) of another image. These assume that the statistics of shallower layers a trained CNN encode style, while deeper layers encode content. Seminal techniques are based on an optimization- pair of content and style images to a CNN encoder and optimizing over randomly initialized image produce activations with similar statistics to the style image at shallower layers and similar activations to the content image at deeper ones (Gatys et al., 2015). This way, stylized image is generated, comprising the semantic and texture attributes of interest.
approach, passing a
While optimization-basee meth generate a diverse stylization due to a random image initialization, au- toencoding methods utilize an image decoder to efficiently stylize arbitrary image pairs on a single forward pass. In what follows, we introduce the autocncoding Style Transfer tochnique adopted by our proposod method due on its generalization and efficiency properties. For an in-depth survey of Style Transfer meth- ods, refer to Jing et al. (2019).
4

---

Published in Transactions on Machine Learning Research (11/2024)
Fast Style Transfer. Dumoulin et al. (2017) proposed an arbitrary Style Transfer method with remarkable generalization properties. Their algorithm, Fast Style Transfer, accurately represents unseen artistic styles by training a model to predict first and second moments of latent image representations at multiple scales. Such moments are used as arguments of a special form of instance normalization, denominated conditional inslance normalization (CIN), to impose style over arbitrary input images.
Given a content image ie E X and a style image ig RCXH,xW, Fast Style Transfer produces a stylized image ies that corresponds to
i = T(ie, RCCH..W.
(4)
where is a stylization network and 2, = F(i.) RD an embedding extracted from the style image via feature extractor F, e.g., InceptionV3 (Szegedy et al., 2016).
We assume 2, to be contracted embedding of the style image (D < C.,W.). The stylization net- work T is comprised by L blocks {41)=r- T extracts high-level features from the content image, aligns them to the style embedding 2, and maps the resulting features to the pixel domain. The style of i, encapsulated in 2, is transferred to the content image using CIN. This is applied to a particular set of layers to impose the target texture and color scheme by aligning feature maps at different scales.
We define the set of layers where CIN is applied A. The normalization imposed via CIN consists of an extended form of instance normalization where the larget mean and standard deviation are extracted from a style representation z.
Interpolation Factor (3)
Given an input RCx and style represen- fation E RD, CIN is defined
= CIN(L, z) E RCxH
(5)
Blending Factor (a)
((h)
1 - (2)
300) (2)
(6)
a(7(3))
where j(6), k E (I....,C) C} corresponds to the k-th input channel, and the sample mean B[2(*) and stan- dard deviation (f((1) are computed along its spatial support. Here, 7(h), (()) RD + I are trainable functions that predict scaling and offset values from the latent representation z for the k-th input chan- nel. The layers in T are characterized by
De,1 = CTN (1((-11))))
1EA A 1 1 A
(7) Figure 2: Feature blending and image interpo- lation. fine-grained control over the final stylized image obtained via interpolation and blending fac- tors a and that operate in the feature and pixel domains. This prevents augmented views from losing semantic information due strong transformations.
t(((e.-1),
where the input of the stylization network corre- sponds to Pr.0 = é The subscript I in the CIN operation indicates that each layer has its own y and X functions to normalize features independently.
Our method uses the Fast Style Transfer algorithm. Given its generalization properties and low-dimensional style representations, it is good match for our framework, where style representations from multiple domains must be efficiently extracted, manipulated and transferred.
4 Proposed Method: SASSL
We provide a detailed description of our SSL augmentation technique. First, we break down SASSL's key components and hyperparameters. Then, we tackle the problem of making the augmented images more diverse by utilizing style references from different. domains in efficient manner.
5

---

Published in Transactions on Machine Learning Research (11/2024)
Algorithm 1: Style transfer augmentation block
Content Image
Style (In-batch)
Style (Extermad)
Stylized (In-batich)
Stylized (External)
Input: io, F, Olmin, Clmaxx Smin: Pmax
Output: tes
< F(bc); # Style representation of content image
2, - ;(i,); # Style representation of style image
U(amiu: amax)
# Blending factor
2 4- (1 a3zz + azs:
# Feature blending
Figure 3: Style Transfer examples. Views gen- erated using style references from the same domain (in-batch) as well as other domains (ezternal). Styl- ization obtained using a blending factor a = 0.5.
3cs f- T(dc, 2):
3 U (Bmin:: (rnax)
Interpolation factor # Stylized image
ies 6- (1 B) + Bess
Style Transfer as data preprocessing. We incorporate Style Transfer to the default preprocessing pipeline of SSL methods. is worth noting that SASSL is not specific to a particular SSL approach, and can be readily applicd with different methods. Figure shows an example of our augmentation pipeline, where Style Transfer is applied after random cropping. A raw input image in is cropped, producing a view that is taken as the content image &z- Given an arbitrary style image i, (we discuss the choice of i, below), the Style Transfer block generates stylized image ics by imposing the texture of i, over ic. Finally, the stylized image ics is passed to the remaining augmentation blocks to produce an augmented sample iaug.
As discussed in recent work on SSL augmentation (Han et al., 2022; Chen et al., 2020a), adding a strong transformation to a self-supervised method tends to degrade performance. For this reason, is crucial to control the amount of stylization imposed in the augmentation stage. We do so by introducing three hyperparameters: probability p [0, 1], which dictates whether an image stylized or not, a blending factor E [0, 1] to combine content z. and style 2, representations, and an interpolation factor B [0, 1] to combine content &e and stylized ie images.
Given style reprosentations extracted from content and style images 2. = F(te) and 2, F(i.), respectively, we obtain an intermediate stylized image ic by applying a convex combination based on blending factor a.
i T(io, 2), 2 (1 a) + az
(8)
Then, the final stylization output is obtained via a convex combination between the intermediate stylized image des and the content image ie based on interpolation factor B.
is (1 3) + Pi..
(9)
Algorithm/ 1 describes our proposed Style Transfer data augmentation block. Figure 2 illustrates the effect of the feature blending and image interpolation operations, showcasing their importance to control the stylization effect without degrading the semnantic altributes.
SASSL operates over minibatches, allowing efficient data pre-processing. Let Ie RBXCxH&XW. and La E RBXCXH,XW be content and style minibatches, respectively, comprised by B images I!!²) and I("), b e [1....,B] B}. Then, the stylized minibatch Ic E BBxCxH,xW. is generated by applying Style Transfer between sample from the content batch and a sample from the style batch, given an arbitrary selection criterion. We propose two alternatives for selecting style images to balance between angmentation diversity and efficiency.
Diversifying style references. In contrast to traditional data augmentation, Style Transfer can leverage second dataset to extract style references. This opens the possibility of selecting style images from different domains, diversifying the transformations applied to the pretraining dataset. SASSL relies on two approaches for sampling style references: external and in-batch stylization.
External stylization consists on pre-computing representations of an arbitrary style dataset and sampling from them during pretraining. This allows controlling the styles to impose the augmented views while
6

---

Published in Transactions on Machine Learning Research (11/2024)
reducing the computational overhead of Style Transfer. Under this configuration, the Style Transfer block receives content minibatch along with minibatch of pre-compated style representations extracted from an arbitrary style dataset, and generates a minibatch of stylized images.
On the other hand, in-batch stylization uses the styles depicted in the content dataset itself by using other images of the content minibatch as style references. This of particular interest for large-scale pretraining datasets covering multiple image categories and thus textures (e.g. ImageNet). So, enabling the use of single dataset for both pretraining and stylization is a valid alternative.
Following this, samples from the same minibatch can be used style references by associating pairs of images in a circular fashion. More precisely, a style minibatch I, E BB is generated by applying a circular shift on the content minibatch indices
( ((bbbb)) bn) mod B)
(10)
where mod denotes the modulo operation and bo is an arbitrary offset. Figure 3 shows ImageNet samples styl- ized using pre-computed style representations from the Painter by Numbers dataset (Kan, 2016) via external stylization, as well as using her ImageNet samples taken from the content minibatch via in-batch stylization.
5
Experiments
5.1 Downstream Task Performance
Table 1: SASSL + MoCo v2 downstream classifi- cation performance on ImageNet. Linear probing accuracy (%) of ResNet-50 backbone pretrained us-
We evaluate the downstream ImageNet classifica- tion accuracy of SST models pretrained via SASST. on the MoCo framework. We compare a MoCo v2 model pretrained with our data augmentation vs. MoCo v2 baseline with default augmentation (Chen et al., 2020b). Note that MoCo v2 and SimCLR use the same loss, architecture, and augmentations (they differ by MoCo's momentum encoding).
ing SASSL + MoCo v2. Mean accuracy reported over five random trials.
Method
Top-1 Acc.
Top-5 Acc.
MoCo v2 (Default)
72.55
91.19
SASSI. MoCo v2 (Ours)
74.64
91.68
Pretraining settings. Our pretraining setup similar to the canonical SSL setup used to pretrain SimCLR and BYOL. We use the same loss, architecture, optimizer, and learning rate schedule as MoCo v2 for fair comparison. We pretrain ResNet-50 encoder on ImageNet for 1, 000 epochs via SASSL. To measure downstream accuracy, add a linear classification head on top of the pretrained backbone and train in a supervised fashion on ImageNet.
SASSL pretraining applies Style Transfer only to the left view (no changes in augmentation are applied to the right view). It applied with a probability p = 0.8 using blending and interpolation factors drawn from uniform distribution a, 2(0.1, 0.3). We found that this modest stylization best complimented the existing augmentations, avoiding rong transformations that hinder performance (Han et al., 2022; Chen et al., 2020a).
Results. Table compares the downstream classification accuracy obtained by our SASSL augmentation approach on MoCo v2 using external stylization from the Painter by Numbers dataset. Results indicate our proposed augmentation improves downstream task performance by 2.09% top-1 accuracy. This highlights the value of Style Transfer augmentation in self-supervised training, where downstream task performance significantly boosts by incorporating transformations that decouple content and style. We also report results with bntcatch stylization in Section 5.5.
5.2 Transfer Learning Performance
To better understand the robustness and generalization of image representations learned using our proposod data augmentation approach, we evaluate its transfer learning performance across various tasks. By in- corporating Style Transfer, we hypothesize that the learned representations become invariant to changes
7

---

Published in Transactions on Machine Learning Research (11/2024)
Table 2: Transfer learning performance. Downstream top-1 classification accuracy (%) of SASSL + MoCo v2 pretrained on ImageNet. Our data augmentation method generates specialized image represen- tations that improve transfer learning performance, as shown in linear probing and fine-tuning scenarios. Mean accuracy reported over five random trials.
Target Datasel
ImageNet ImageNNt-11% iNat21 Retinopathy DTD Food101 CIPARIO CIFARIOD SUN397
Caltech-101 Flowers
Lineor Probing
None (Default)
72.55
53.23
11.33
75.88
68.68
73.82
89
71.93
69.96
53.15
88.19
93.39
ImageNet (Ours]
74.07
56.87
75.75
74.43
90.93
69.67
64.87
INat21 (Ours)
74.28
56.76
44.70
75.75
72.75
T4.3
91.04
73.29
70.07
63.96
89.89
94.7
Retinopathy (Ours)
74.02
56.99
44.9
75.78
73.73
74.53
90.8
73.3
69.63
64.06
89.17
94.94
Style Datasel
DTD (Ours)
74.32
56.77
45.08
75.76
74.41
74.88
91.04
73.41
69.71
64.58
80.3
95.24
PBN (Ours)
74.64
56.9
45.02
75.79
72.77
74.37
90.85
73.38
69.69
64.12
89.59
95.45
Fine-buninng
None (Default)
74.89
61.61
77.92
78.89
71.54
87.25
96.91
83.4
74.25
83.63
89.27
95.75
ImageNet (Ours]
75.52
51.74
79.64
87.48
97.0
73.89
INat21 (Ours)
75.58
51.86
79.19
79.6
71.35
87.4
97.05
83.29
74.05
90.04
88.55
95.76
Retinopathy (Ours)
75.52
51.76
79.23
79.63
72.07
87.39
96.97
83.68
74.26
89.96
88.44
96.34
DTD (Ours)
73.24
51.73
79.24
79.7
70.59
87.66
96.77
83.28
74.17
89.59
89.54
95.59
PBN (Ours)
75.05
51.85
79.2
79.63
71.33
87.56
96.97
83.36
74.18
89.75
88.97
95.77
in appearance such as color and texture. This forces the feature extraction process to rely exclusively on semantic attributes. As a result, the learned representations may become more robust to domain shifts, improving downstream task performance across datasets. We empirically show this property by evaluating the transfer learning performance of image representations trained using SASSL under lincar probing and fine-tuning scenarios.
Downstream settings. We compare the transfer learning accuracy of ResNet-50 pretrained via MoCo v2 using SASSL against MoCo v2 baseline with default data augmentation. The evaluated models are pretrained on ImageNet and transferred to eleven target datasets: ImageNet-1% subset (Chen et al., 2020a), iNaturalist '21 (iNat21) (iNaturalist 2021), Diabetic Retinopathy Detection (Retinopathy) (Kaggle & Eye- Pacs, 2015), Describable Textures Dataset (DTD) (Cimpoi et al., 2014), Food101 (Bossard et al., 2014), CIFAR10/ 100 (Krizhevsky, 2009), SUN397 (Xiao et al., 2010), Cars (Krause al., 2013), Caltech-100 101 (Fei-Fei et al., 2004), and Flowers (Nilsback & Zisserman, 2008).
To have clear idea of the effect of the style dataset in SASSL's pipeline, we pretrain five ResNet-50 backbones, each using a different style. We use ImageNet, iNat21, Retinopathy, DTD, and Painter by Numbers (PBN) as style datasets. More precisely, we transfer five models, each pretrained on a different style, to each of cleven target datascts. We also include ImageNet as target datasct to compare the effect of different styles on downstream task performance. This leads to 60 transfer learning scenarios used to better understand the effect of various styles on different image domains.
Transfer learning is evaluated in terms of top-1 classification accuracy on linear probing and fine-tuning. All models were pretrained as described in Section 5.1. We report mean accuracy across five trials. Please refer to Appendix A.4 for full linear probing and fine-tuning training and testing settings.
Results. Table 2 shows the top-1 classification accuracy obtained via transfer learning. For linear probing, SASST significantly improves the average performance on eleven out of twelve target datasets by up to 10% top-1 classification accuracy. For Retinopathy, SASSL obtains on-par lincar probing accuracy the default MoCo v2 model.
For fine-tuning, all models trained via SASSL outperform the baseline. Results show the average top-1 classification accuracy improves by up to 6%. This suggests SASSL generalizes across datasets, spanning from textures (DTD) to medical images (Retinopathy). Note that, for a fair comparison, we do not perform hyperparameter tuning. Interestingly, the relative performance obtained from different style datasets gener- ally differs comparably to the mcasurement uncertainty, which is shown for these experiments in Scction A.5 of the supplementary material. This suggests that the choice of style dataset is secondary in importance, while the main benefit comes from the of SASSL itself.
8

---

Published in Transactions on Machine Learning Research (11/2024)
5.3 Few-shot Learning Performance
Table 3: SASSL + MoCo v2 Few-shot learning performance. One and ten-shot top-1 classification accuracy (%) of representations learned via SASSI.
To further demonstrate the representation learning capabilities of the data augmentation imposed via SASSL, conduct experiments on few-shot classi- fication. We compare trained via SASSL + MoCo v2 against MoCo v2 baseline in the context of one and ten-shot learning on ImageNet.
MoCo v2. Accuracy reported single trial.
ResNet-50 backbone pre-
Method
One-shot Acc. Ten-shot Acc.
MoCo v2 (Default)
19.56
45.05
SASSL + MoCo v2 (Ours)
20.55
46.73
Table shows the few-shot classification accuracy. Results reveal that SASSL boosts few-shot classification top-1 accuracy by over 1% in both one and ten-shot learning. This aligus with our previous experiments, suggesting that SASSL promotes more general image representations.
5.4 Additional Downstream Performance Evaluation
Performance other SSL methods. To assess SASSL's broader impact, evaluate its effectiveness two other SSL methods, SimCLR and BYOL. We pretrain ResNet-50 backbones with each method, and then use linear probing on ImageNet to compare the quality of their learned represenlations. For each method, default pretraining and linear probing configurations are used. For SASSL, we employ its recommended hyperparameters (a, 3 E 00.1, p = 0.8) and PBN as style datasot.
Table 4 shows the accuracy altained by SimCLR and BYOL equipped with SASSL Results show our proposed data augmentation technique boosts top-1 accuracy by approximately 1% in both cases, highlighting its potential across multiple SSL techniques.
Table 4: SASSL downstream performance us- ing alternative SSL methods. Linear probing accuracy (%) on ImageNet using ResNet-50 back- bone pretrained via SimCLR and BYOL. Accuracy reported on a single trial.
Table 5: SASSL + MoCo downstream per- formance using alternative backbones. Linear probing accuracy (%) on ImageNet using ResNet-50 (x4) and ViT-B/16 representation models. Accu- racy reported on single trial.
Method
Top-1 Acc.
Top-5 Acc.
Backbone
Method
Top-1 1 Acc. Top-5 Ace.
SimCLR (Default)
68.62
88.7
ResNet-50
MoCa v2 (Default)
77.2
93.32
SASSL + SimCLR (Ours)
69.58
89.01
x4 (375M) SASSL + MoCo v2 (Ours)
78.21
93.98
BYOL (Default)
74.09
91.83
BIT-B/16
MoCo v3 (Default)
75.01
92.43
SASSL BYOL (Ours)
75.13
92.12
(86M) SASSL | MoCo v3 (Ours)
75.51
92.56
SwAV (Default)
70.45
89.6
SASSL SwAV (Ours)
71.3
90.39
Performance on other representation models. We explore SASSL's performance models with varying complexity and architecture. For complexity, employ ResNet-50 (x4), a scaled-up version of the previously evaluated ResNet-50 (from 24 to 375 million parameters). This allows us to probe how SASSL scales with increased model size. In terms of architecture, we employ VIT-B/16, a Transformer--ased backbone with 86 million parameters and a distinct design compared to previous CNN models.
We pretrain and lincarly probe a ResNet-50 (x4) representation model on ImageNet MoCo v2. Protraining and downstream settings follow our default configuration, as documented in the Appendices A.3 and A.4. Similarly, we pretrain and linear probe ViT-B/IK 116 model on ImageNet via MoCo v3. In this case, SASSL employed a blending factor a uniformly sampled between 0.1 and 0.5.
Table 5 reports the rownst.ream classification accuracy for ResNet-50 (x4) and ViT-B/16. ResNet-50 (x4) results show SASSL improves top-1 classification accuracy by 1.1%, mirroring its earlier improvement. Sim- ilarly, ViT-B/16 results show SASSL improves top-1 accuracy by 0.5%. These suggest that SASSL is not limited CNN backbones, but can also be extended to ViTs. While this margin currently smaller for ViTs, emphasize that no hyperparameter tuning was employed in these experiments.

---

Published in Transactions on Machine Learning Research (11/2024)
5.5 Ablation Studies
To shed light on how SASSL affects accuracy on ImageNet, we break down its components and assess individual contributions to downstream performance. We also study how aligning different layers in the stylization network T boosts accuracy. See Appendices A.6 and A.7 for additional ablations and SASSL's computational requirements.
SASSL components. For the ablation study, we cover four cases: (i) MoCo v2 with default augmentation, (ii) SASSI, + MoCo v2 using in-batch representation blending and no pixel interpolation (3 = 1), (iii) SASSL + MoCo v2 using in-batch representation blending and pixel interpolation, and (iv) SASSL + MoCo using all its attributes (blonding, interpolation and an external style dataset).
Table 6 shows our ablation study using MoCo v2 as SST technique. Results highlight. the importance of controlling the amount of stylization using both representation blending and image interpolation. With- out image interpolation, using Style Transfer as data augmentation degrades the downstream classification performance by more than 1.5% top-1 accuracy.
On the other hand, by balancing the amount of stylization via blending and interpolation, SASSL boosts performance by more than 1.5%. This is a significant improvement for the challenging ImageNet scenario. Finally, by incorporating an external style dataset such as PBN, we further improve downstream task per- formance by almost 2.1% top-1 accuracy. This shows the importance of diverse style references and their effect on downstream lasks.
Table 6: Ablation study. Linear probing accuracy (%) for representations learned via SASST under dif- ferent configurations. Mean accuracy reported over five random trials.
Table 7: Effect of the number of stylized lay- ers in downstream performance. Linear prob- ing classification accuracy (%) of ResNet-50 model pretrained via SASSL + MoCo v2, where Style Transfer is applied using subset of the available layers. Accuracy reported on a single trial.
Method
Configuration
Style
Top-1 Aoe. Top-5 Ace.
MoCo (Default)
72.55
91.19
Method
Stylized Layers Top-1 Acc.
Top-5 Acc.
p 0.8, é (0.1,0.3] ==1
ImageNet (in-botch)
70.87
89.33
MoCo v2 (Default)
72.97
90.86
None (2 ze)
73.77
91.64
MoCo v2 (Ours)
SASSL
é 0.3]
p
ImageNet (in-bbteck)
74.07
91.58
SASSL + MoCo v2 (Ours)
First 4 layers
73.75
91.58
6 0.3)
First 8 layers
74.09
91.76
p 0.8,
PBN (external)
First 10 layers
74.27
91.74
E 00.1
74.64
91.68
8. [0.1,
All (13 layers)
75.38
92.21
Number of stylized layers. We explore how the number of layers used to apply style transfer via CIN affects downstream performance. We analyze three cases: (i) stylizing using the first two residual blocks of the Stylization Network T (four layers from blocks 1 and 2), (ii) the first four residual blocks (eight layers from blocks 1 to 4), and (iii) all five residual blocks (ten layers).
For each case, we pretrain and linearly probe a ResNet-50 on ImageNet using SASSL + MoCo v2 with its recommended settings (a, 3 [0.1, p = 0.8) and PBN as style dataset. To fully remove the effect of a style embedding 2., our comparison includes a model pretrained using the content image itself as style reference (2 2.). We also compare our full SASSL MoCo v2 model, stylizing all residual and upsampling blocks of T (thirteen layers).
Table 7 shows progressive cnhancement in accuracy with incrcasing stylization depth. Adding stylization to the first. four layers showed negligible gains, mirroring the accuracy of the unaligned model. Stylizing the first eight and ten layers yielded modest improvements of 0.34% and 0.52%, respectively, implying a growing influence of deeper layers on accuracy. Notably, pretraining with full stylization, encompassing both residual and upsampling layers, attains 1.61% accuracy boast, suggesting the importance of aligning deeper upsampling layers for downstream performance.
10

---

Published in Transactions on Machine Learning Research (11/2024)
6 Conclusion
We propose SASSL, novel data augmentation approach based on Neural Style Transfer that exclusively transforms the style of training samples, diversifying data augmentation during pretraining while preserving semantic attributes. We empirically show our approach outperforms well-established methods such as MoCo v2, SimCLR and BYOL by up to 2% top-1 classification accuracy on ImageNet. SASSI. also improves the transfer capabilities of learned representations, enhancing linear probing and fine-tuning performance across domains by up to 10% and 6% top-1 accuracy, respectively. Our technique can be extended to other SSL, methods and models with minimum hyperparameter changes, experimentally shown.
Broader Impact Statement
This work proposes novel data augmentation approach leveraging Neural Style Transfer to enhance Self- supervised Learning, particularly for domains with limited data or expensive annotations. Our method utilizes semant ic-wwaee image preprocessing to extract robust representations that generalize across diverse domains. This advancement tackles the critical challenge of using unlabeled data for Deep Learning, which has many potential positive impacts both technical and socictal fronts. SASSL's style transfer component helps reduce sensitivity lo image texture, potentially improving model robustness texture bias. However, since method primarily modifies data augmentation, it may not fully address other potential biases arising from pre-training datasets or learning strategies.
Acknowledgements
The authors would like to thank Arash Afkanpour and Luyang Liu for their insightful comments and feed- back.
References
Martin Abadi, Paul Barham, Jianmin Chen, Zhileng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, Manjunath Kudlur, Josh Levenberg, Rajat Monga, Sherry Moore, Derek G. Murray, Benoit Steiner, Paul Tucker, Vijay Vasudevan, Pete Warden, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. Tensorflow: A system for large-scale machine learning. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDT 16), pp. 265 283, 2016. URL https: //www//wwwwww//// usenix. org/ wttps//siteee//e//////a/ss www.feeeenneosssit//ssiiiiiiiiee
pdf. 18
Mahmoud Assran, Mathilde Caron, Ishan Misra, Piotr Bojanowski, Florian Bordes, Pascal Vincent, Armand Joulin, Mike Rabbat, and Nicolas Ballas. Masked siamese networks for label-officient learning. In European Conference Computer Vision, pp. 456-473. Springer, 2022. 2,
Yingbin Bai, Erkun Yang, Zhaoqing Wang, Yuxuan Du, Bo Han, Cheng Deng, Dadong Wang, and Tongliang Liu. Rsa: Reducing semantic shift from aggressive augmentations for self-supervised learning. Advances
in Neural Information Processing Systems, 35:21128 21141, 2022. 3
Adrien Bardes, Jean Ponce, and Yann LeCun. Vicreg: self-supervised learning. arXiv preprint arXin:2105. 04906, 2021.
regularization for
Lukas Bossard, Matthieu Guillaumin, and Lue Van Gool. Food-101 mining discriminative components with random forests. In European Conference on Computer Vision, 2014. 8, 16
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsu- pervisod learning of visual features by contrasting cluster assignments. Advances in neural information processing systems, 33:9912 9924, 2020. 2,
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 9650 9660, 2021. 2,
11

---

Published in Transactions on Machine Learning Research (11/2024)
Haibo Chen, Zhizhong Wang, Huiming Zhang, Zhiwen Zuo, Ailin Li, Wei Xing, Dongming Lu, et al. Artistic style transfer with internal-external learning and contrastive learning. Advances in Newral Information Processing Systems, 11885611-267727 2021a. 3
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive lcarning of visual representations. In International conference on machine learning, pp. 1597-1607. PMLR, 2020a. 2, 3, 6, 7, 8, 16, 18, 19
X. Chen, S. Xie, and K. He. An empirical study of training self-supervised vision transformers. In 2021 IEEE/CVP International Conference Compuler Vision (ICCV), pp. 9620 9629, Los Alamilos, CA, USA, oct 2021b. IEEE Computer Society, doi: 10.1109 ALLL00/1CCCA892222222..0000 1eeecomputersociety,
URL https://doi.
/1tp///..1///C/4492222 2021.
202.0222..221.0090 00950. 4
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 15750 15758, 2021. 2, 4
Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive
lcarning. arXiv preprint arXiv:2003.042: 2020b. 4, 7
M. Cimpoi, S. Maji, I. Kokkinos, S. Mohamed, and A. Vedaldi. Describing textures in the wild. In Proceedings of the IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2014. 8, 16
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pallern recognition, pp. 248 255. Ieee, 2009. 2, 16
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional
transformers for language understanding. arXiv preprint urXiv:1810.4480) 2018. 2
Vincent Dumoulin, Jonathon Shiens, and Manjunath Kudlur. A learned representation for artistic style. International Conference on Learning Representations, 2017. URL https: htt.://rrrrrreeooeo//oorooricicc net/forumidd= BJO-BuTig.
Li Fei-Fei, Rob Fergus, and Pietro Perona. Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. Computer Vision and Pattern Recognition Workshop, 2004. 8, 16
Leon A Gatys, Alexander S Ecker, and Matthias Bethge. neural algorithm of artistic style. arXiv preprint arXiv:1508.068: 06576, 2015.
Lcon A Gatys, Alexander S Ecker, and Matthias Bethge. Image style transfer using convolutional neural norworks. In Proceedings of the IEEE Conference on Computer Vision and Pallern Recognition, pp. 2414 2423, 2016. 3
Leon A Gatys, Alexander S Ecker, Matthias Bethge, Aaron Hertzmann, and Eli Shechtman. Controlling perceptual factors in neural style transfer. In Proceedings of the IEFF Conference on Computer Vision and Pattern Recognition, pp. 3985-3993, 2017.
Robert Geirhos, Patricia Rubisch, Claudio Michaclis, Matthias Bethge, Felix A Wichmann, and Wicland Brendel. Imagenet-trained cnns are biased towards texture; increasing shape bias improves accuracy and robustness. arXiv preprint arxiv:1811 12231, 2018.
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. Advances in neural information processing systems, 33:21271 21284, 2020. 2, 4, 18, 19
Junlin Han, Pengfei Fang, Waihao Li, Jie Hong, Mohammad Ali Armin, Ian Reid, Lars Petersson, and Hong- dong Li. You only cut once: Boosting data augmentation with a single cut. In International Conference on Machine Learning, pp. 8196 8212. PMLR, 2022. 6,
12

---

Published in Transactions on Machine Learning Research (11/2024)
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEFE Conference on Compuder Vision and Pattern Recognition. (CVPR), June 2016.
18
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervisod visual representation learning. In Proceedings of the IEEE/CVP conference on computer vision and pattern recognition, pp. 9738, 2020. 2
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 16000-16009, 2022. 2
David J Heeger and James R Bergen. Pyramid-based texture
In Proceedings of the
Conference on Computer Graphics and Interactive Techniques, pp. 229-238, 1995. 3, 21
Eric Heitz, Kenneth Vanhoey, Thomas Chambon, and Laurent Belcour. sliced Wasserstein loss for neu- ral texture synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9412 9420, 2021.
Minui Hong, Jinwoo Choi, and Gunhee Kim. Stylemix: Separating content and style for enhanced data augmentation. In Proceedings of the IEEE/CVF conference on compuler vision and pattern recognition, pp. 14860, 2021. 3
Xun Huang and Serge Belongie. Arbitrary style transfer in real-time with adaptive instance normalization. In Proceedings of the IEEE International Conference Computer Vision, pp. 1501-15100 2017. 3
iNaturalist 2021. iNaturalist 2021 competition dataset.
tree//natter/2021, 2021. 8, 16
Philip TG Jackson, Amir Atapour Abarghouei, Stephen Bonner, Toby P Breckon, and Boguslaw Obara. Style augmentation: data augmentation via style randomization. In CVPR workshops, volume 6, pp. 10-11, 2019. 3
Yongcheng Jing, Yezhou Yang, Zunlei Feng, Jingwen Ye, Yizhou Yu, and Mingli Song. Neural style transfer: A review. IEEE transactions on visualization and computer graphies, 26(11):3365 3385, 2019.
Yongcheng Jing, Xiao Liu, Yukang Ding, Xinchao Wang, Errui Ding, Mingli Song, and Shilei Wen. Dynamic instance normalization for arbitrary style transfer. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 1369,4376, 2020. 3
Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual losses for real-time style transfer and super- resolution. In European conference computer vision, pp. 694-711. Springer, 2016. 21
Kaggle and EyePacs. Kaggle diabetic retinopathy detection, jul 2015. URL https://wwet btttps:/www..kaggle.co
Saabeii--eeiinppttyyydeettttttad 8, 16
Wendy Kan. Painter by numbers, 2016. URL httpppp/aaaggle..c////mettttttta
7, 16
Agnan Kessy, Alex Lewin, and Korbinian Strimmer. Optimal whitcning and decorrelation. The American Statislician, 72(4):309 314, 2018.
Simon Kornblith, Jonathon Shlens, and Quoc V Le. Do better imagenet models transfer better? In Proceedings of the IEEE/CVF conference computer vision and poltern recognition, pp. 2661-2671, 2019. 19
Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fino-grained catego- rization. In 4th International IEEE Workshop on 3D Representation and Recognition (3dRR-13) 13), Sydney, Australia, 2013. 8, 16
13

---

Published in Transactions on Machine Learning Research (11/2024)
Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, 2009. 8, 16
Hankook Lee, Kibok Lee, Kimin Lee, Honglak Lee, and Jinwoo Shin. Improving transferability of represen- tations via augmentation-avare self-superviiion. Advances in Neural Information Processing Systems, 34: 17710-17722, 2021. 3
Yanghao Li, Naiyan Wang, Jiaying Liu, and Xiaodi Hou. Demystifying neural style transfer. arXiv preprint 1eXi::101101000 2017a. 3
Yijun Li, Chen Fang, Jimei Yang, Zhaowen Wang, Xin Lu, and -isag- Yang. Universal style transfer via feature transforms. Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS'17, pp. 385 395, Red Hook, NY, USA, 2017b. Curran Associales Inc. ISBN 9781510860964.
Songhua Liu, Tianwei Lin, Dongliang He, Fu Li, Meiling Wang, Xin Li, Zhengxing Sun, Qian Li, and Errui Ding. Adaatia: Revisit attention mechanism in arbitrary neural style transfer. In Proceedings of the IEEE/CVP International Conference Computer Vision, pp. 1449-6558, 2021.
M-E. Nilsback and A. Zisserman. Automated flower classification over large number of classes. In Pro- ceedingg of the Indian Conference on Computer Vision, Graphics and Image Processing, Dec 2008, 8, 16
Javier Portilla and Fero P Simoncelli. A parametric texture model based on joint statistics of complex
wavelet coefficients. International Journal of Computer Vision, 40(1):49 70, 2000. 3, 21
Senthil Purushwalkam and Abhinav Gupta. Demystifying contrastive self-supervised learning: Invariances, augmentations and dataset biases. Advances in Newral Information Processing Systems, 13:3407-3415 2020.
Colorado Reed, Scan Metzger, Aravind Srinivas, Trevor Darrell, and Kurt Keutzer. Selfaugment: Auto- matic augmentation policies for self-supervised learning. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pp. 2674 2683, 2021.
Eric Risser, Pierre Wilmot, and Connelly Barnes. Stable and controllable neural texture synthesis and style transfer using histogram losses. 2017. URL
http://axwvvvooorab/reeese/ 08893.
Lu Sheng, Ziyi Lin, Jing Shao, and Xiaogang Wang. Avatar-net: Multi-scale zero-shot style transfer by feature decoration. In Proceedings of the TEFE Conference Computer Vision and Pattern Recognition, pp. 8242 8250, 2018. 3
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint ar Xic::11499 1556, 2014. 3
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pallern recognilion, pp. 2818 2826, 2016. 5
Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, and Phillip Isola. What makes for good views for contrastive learning? Advances in neural information processing systems, 33:6827-6839, 2020.
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, ((11), 2008. 17
Diane Wagner, Fabio Ferreira, Danny Stoll, Robin Tibor Schirreesseer, Samuel Müller, and Frank Hutter. On the importance of hyperparameters and data augmentation for self-super learning. arXiv preprint arXiv:2207.07885,, 2022. 3
14

---

Published in Transactions on Machine Learning Research (11/2024)
Zhizhong Wang, Lei Zhao, Haibo Chen, Lihong Qiu, Qihang Mo, Sihman Lin, Wei Xing, and Dongming Lu. Diversified arbitrary style transfer via deep feature perturbation. In Proceedings of the IEEE/CVP Conference on Computer Vision and Pattern Recognition, pp. 7789 7798, 2020. 3
J. Xiao, J. Hays, K. A. Fhinger, A. Oliva, and A. Torralba. Sun database: Large-scale scene recognition from abbey to zoo. In 2010 IEEE Computer Society Conference Computer Vision and Pattern Recognition. pp. 3485 3492, June 2010. doi: 10.1109/ (011099/VP.221101 2VPP.2010.5539970) 8, 16
Jaejun Yoo, Youngjung Uh, Sanghyuk Chun, Bycongkyu Kang, and Jung- Woo Ha. Photorcalistic style transfer via wavelet. transforms. In Proceedings of the IEEE International Conference on Computer Vision, pp. 9036-9045, 2019.
Yang You, Igor Gitman, and Boris Ginsburg. Large batch training of convolutional networks. arXiv preprint arXiv:1706 778.03888 2017. 18
Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self-supervised learning via redundancy reduction. In International Conference on Machine Learning, pp. 12310 12320. PMLR, 2021.
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 586 595, 2018. 21
Xu Zheng, Tejo Chalasani, Koustav Ghosal, Sebastian Lutz, and Aljosa Smolic. Stada: Style transfer data augmentation. arXiv preprint ww.X::1109.01000 2019.
Song Chun Zhu, Xiu Wen Liu, and Ying Nian Wu. Exploring texture ensembles by efficient Markov chain monte carlo-toward "trichromacy" theory of texture. IEEE Transactions on Pattern Analysis and Ma- chine Intelligence, 22(6):554 569, 2000. 3, 21
15


Reviews:
[{"review_id": "TOubrn0g5z", "reviewer": "Reviewer_w3Rd", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**\n- The paper is well-written and easy to follow, with all of the details presented in an easy-to-read way.\n- The idea of using neural style transfer as an additional data augmentation technique is simple and compelling. Another compelling feature is that NST as data augmentation can effectively be an additional add-on to any standard self-supervised learning method.\n- Across many settings, there are clear performance gains.\n\n**Weaknesses:**\n- There are a number of hyperparameters one could choose in this method, making it potentially difficult to use right out of the box. The main body of the paper could be improved with a deeper discussion of these choices. For example, a discussion on the choice of style for a given task and dataset is presented in the appendix, but I feel as though it would be more beneficial to have this in the main text. Please see the requested changes section for more.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work introduces a novel data augmentation method for self-supervised learning using neural style transfer. Most methods for self-supervised learning augment data by incorporating simple transformations such as random cropping, flips, noise addition, and more. Such transformations could destroy semantic information in the image, leading to potential degradation in downstream performance. This work considers using neural style transfer as a technique to augment data in such a way that semantic information is not lost while also allowing for effective data augmentation. Neural style transfer morphs a content image to a particular style using the features of a deep neural network. For faster generation of augmented images, the authors consider autoencoder-based approaches to generate augmented images, as opposed to optimization-based approaches to generate such images. Once generated, these images can then be used in any self-supervised learning training pipeline. The method is shown to improve performance over standard self-supervised methods in image classification along with aiding in transfer learning.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I believe addressing the following would aid in strengthening the work in my view:\n\n**Main change**\n- Could the authors move the discussion of the choice of style for improved performance in the main body? \n\n**Minor questions**\n- Do the authors have intuition as to why in certain cases, adding style from the target dataset does not achieve high performance? For example, in Table 2, on ImageNet, other using other datasets for SASSL improved performance over ImageNet. \n- I appreciate the ablation study on the interpolation parameter, which matches my intuition that adding too much style can degrade performance. Is there a similar effect on the blending parameter?", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "rSpOTiUQP7", "reviewer": "Reviewer_hgHu", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n1. **Improved downstream performance**: SASSL boosts top-1 image classification accuracy on ImageNet by up to 2% compared to established SSL methods like MoCo, SimCLR, and BYOL. It also achieves superior transfer learning performance across various datasets.\n2. **Generalization across datasets**: Results show SASSL generalizes across datasets, spanning from textures (DTD) to medical images (Retinopathy), with linear probing accuracy improving by up to 10% and fine-tuning accuracy by up to 6%.\n3. **Effectiveness on different SSL methods**: SASSL boosts top-1 accuracy by approximately 1% when applied to SimCLR and BYOL, showing its potential across multiple SSL techniques.\n4. **Improved few-shot learning**: SASSL learns more general image representations, boosting few-shot classification top-1 accuracy by over 1% in both one and ten-shot learning on ImageNet.\n5. **Flexibility with representation models**: SASSL achieves consistent improvements when applied to different models and architectures, such as ResNet-50 and ViT-B/16.\n\n## Weaknesses\n1. **Selection of stylization:** Although the authors provide in the supplementary a way to determine the similarity between the datasets, it is not stated or presented a way to select the optimal dataset to use as style. it seems one needs to try more than one and look at the test accuracy, defeating the purpose of the test set. according to the t-sne plot in the supplementary, it seems that, apart from the Diabetic Retinopathy dataset, the rest of the datasets are very close in this embedding space, making the choice even more difficult. a more thorough discussion of the selection process would be beneficial.\n2. **Image-to-image translation:** Although the proposed SASSL reaches reasonable results on the proposed benchmark, the paper lacks a discussion of more recent approaches to image stylization/transformation, such as image-to-image generation, which diffusion models are good at. One notable example is InstructPix2Pix [1], where the image can be stylized or transformed by describing via text the desired transformation. While I am not asking to run this pipeline, I would like a discussion of the advantages/disadvantages of the proposed method w.r.t. these approaches.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes SASSL (Style Augmentations for Self-Supervised Learning), a novel data augmentation technique for self-supervised learning (SSL) that leverages Neural Style Transfer to generate semantically consistent augmented samples. SASSL decouples semantic and stylistic attributes in images and applies transformations exclusively to their style while preserving content, resulting in diverse samples that better retain semantic information. The authors show that SASSL boosts downstream classification performance on ImageNet by up to 2% compared to established SSL methods like MoCo, SimCLR, and BYOL, while achieving superior transfer learning results across various datasets.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "My main concern is related to the choice of the stylization dataset, where a more thorough discussion is needed in my opinion. It does not hinder the proposed method's strengths but it limits its applicability.", "broader_impact_concerns": "The paper includes a Broader Impact Statement. The statement could include a description of potential bias learned by the model, especially in self-supervised models, where the bias could be learned implicitly.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "wmSi7oA2pS", "reviewer": "Reviewer_pV6L", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n\n* The paper is well-written and easy to follow. \n* The results have been presented in a clear format so that the reader can understand the gains. \n\n## Weaknesses \n\n* Effects of the proposed method on non-contrastive self-supervised methods like SwAV [1] are missing. \n* Ablations on the important degrees of freedom introduced in the proposed method are missing. \n* Training throughput information is missing. I think this is important to mention because stylization can incur an additional increase in the preprocessing time. \n\n## References\n\n[1] Unsupervised Learning of Visual Features by Contrasting Cluster Assignments, https://arxiv.org/abs/2006.09882.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors investigate the impacts of using stylization in the data augmentation pipeline of contrastive self-supervised learning frameworks like SimCLR, BYOL, and MoCo v2 and report gains over the baseline frameworks.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "* In the abstract, it would be better to give the readers a hint about the changes in training throughput because of stylization (if there are any). My point about including some information on the timing-related ramifications stands for other areas of the paper, such as the introduction, results, etc. Additionally, directly mentioning the gains obtained in the downstream performance in the abstract would be informative. \n\n* Section 2.1 misses the multi-crop augmentation policy introduced in SwAV [1] and is being used in DINO [2] and other works like [3], [4]. \n\n* The preliminaries on SSL could be categorized into contrastive and non-contrastive approaches rather than being restricted to SimCLR only. This gives readers a better view of the field. \n\n* Could include a section studying the implications of stylization in the context of non-contrastive frameworks. \n\n* Some more ablations could be:\n  * Does increasing the probability of stylization during the latter half of the training help improve the performance? It's inspired by the EMA decaying scheme, which is usually followed in works like data2vec [5]. \n  * Why do we need a circular scheme when using the in-batch stylization? \n  * Can we use a blend of in-batch and external dataset stylization? \n  * Can we use an aggregate dataset sourced by combining images from all the external datasets used for stylization? \n  * Why is only MoCo v2 used for reporting downstream performance? \n\n* Details that could be added:\n  * It Could be worth detailing what the \"1%\" metric means in Table 2. \n  * Highlight the importance of using different stylization datasets for different downstream datasets. Table 2 confirms this phenomenon. \n  * The ablation in \"Number of stylized layers\" is not fully clear to me. Does it mean fine-tuning the stylization network along with the base model being pre-trained? \n\n## References\n\n[1] Unsupervised Learning of Visual Features by Contrasting Cluster Assignments, https://arxiv.org/abs/2006.09882. \n\n[2] Emerging Properties in Self-Supervised Vision Transformers, https://arxiv.org/abs/2104.14294. \n\n[3] Self-supervised Pretraining of Visual Features in the Wild, https://arxiv.org/abs/2103.01988. \n\n[4] Vision Models Are More Robust And Fair When Pretrained On Uncurated Images Without Supervision, https://arxiv.org/abs/2103.01988. \n\n[5] data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Language, https://arxiv.org/abs/2202.03555.", "broader_impact_concerns": "Filtering out the style datasets so that they don't include inappropriate or licensed images could be important. The latter could lead to concerns about copyright infringement.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper introduces a data augmentation method for self-supervised learning based on neural style transfer. The stylization is grounded in an autoencoding approach, which maintains efficiency while preserving semantic details through interpolation and blending. Experiments are conducted for image classification on ImageNet and for transfer learning.

The paper initially received positive feedback. The reviewers appreciated the idea of decoupling semantic from style information for data augmentation in self-supervised learning and found the experiments convincing. However, they raised concerns regarding the setting of hyperparameters and the impact of the style dataset on final performance. The authors' feedback was delayed.A fter the discussion period, reviewers Rw3Rd and Rw3Rd recommended acceptance, whereas RpV6L was inclined to reject the paper, mainly because they did not receive a response before making their recommendation.

The AE has carefully reviewed the submission and the discussions. The AE considers that the submission constitutes an interesting contribution to self-supervised learning, as style augmentation can complement the standard augmentations used in the literature. Overall, the authors' rebuttal, although submitted after the discussion period, was convincing regarding hyperparameter settings, training throughput, and the inclusion of new baselines, e.g., SwAV. The discussion on the choice of style datasets was more superficial but will likely inspire future work in this direction. Therefore, the AE recommends acceptance.
Decision: Accept (poster)