Title: RETHINKING DATA AUGMENTATION: SELF-SUPERVISION AND SELF-DISTILLATION

Abstract: Data augmentation techniques, e.g., flipping or cropping, which systematically enlarge the training dataset by explicitly generating more training samples, are effective in improving the generalization performance of deep neural networks. In the supervised setting, a common practice for data augmentation is to assign the same label to all augmented samples of the same source. However, if the augmentation results in large distributional discrepancy among them (e.g., rotations), forcing their label invariance may be too difficult to solve and often hurts the performance. To tackle this challenge, we suggest a simple yet effective idea of learning the joint distribution of the original and self-supervised labels of augmented samples. The joint learning framework is easier to train, and enables an aggregated inference combining the predictions from different augmented samples for improving the performance. Further, to speed up the aggregation process, we also propose a knowledge transfer technique, self-distillation, which transfers the knowledge of augmentation into the model itself. We demonstrate the effectiveness of our data augmentation framework on various fully-supervised settings including the few-shot and imbalanced classification scenarios.

1. INTRODUCTION:

Training deep neural networks (DNNs) generally requires a large number of training samples. When the number of training samples is small, DNNs become susceptible to overfitting, causing high generalization errors on the test samples. This overfitting problem is at the center of DNN research, where many regularization techniques have been investigated in the literature (Srivastava et al., 2014; Huang et al., 2016; Gastaldi, 2017) . The most explicit and easy-to-use regularization technique is data augmentation (Zhong et al., 2017; DeVries & Taylor, 2017; Zhang et al., 2018; Cubuk et al., 2019) , which aims to increase the volume of the training set by altering the existing training data.

In supervised learning scenarios, data augmentation is done simply by augmenting each sample with multiple transformations that do not affect their semantics. Consequently, data augmentation during training forces DNNs to be invariant to the augmentation transformations. However, depending on the type of transformations, learning transformation-invariant properties may be difficult or compete with the original task, and thus could hurt the performance. For example, for certain fine-grained image classification tasks (e.g., species of birds), the color information could be crucial in class discrimination. In such a case, the data augmentation using color transformations should be avoided.

The evidence that learning invariance is not always helpful could be found in many recent works on self-supervised learning (Gidaris et al., 2018; Zhang et al., 2019; Doersch et al., 2015; Noroozi & Favaro, 2016) , where the model is trained with artificial labels constructed by input transformations, e.g., the rotation degree for rotated images, without any human-annotated labels. These works have shown that it is possible to learn high-level representations just by learning to predict such transformations. This suggests that some meaningful information could be lost when trying to learn a transformation-invariant property under conventional data augmentation.

While self-supervision was originally developed for unsupervised learning, there are many recent attempts to use it for other related purposes, e.g., semi-supervised learning (Zhai et al., 2019) , robustness (Hendrycks et al., 2019) and adversarial generative networks (Chen et al., 2019) . They commonly maintain two separated classifiers (yet sharing common feature representations) for the original and self-supervised tasks. However, such a multi-task learning strategy also forces invari- 

c i a x a Z j h 2 E k U E h F y b I f j u 5 n f f k S l W S w f z C T B Q J C h Z B G j x F i p E f V L Z a / i z e G u E j 8 n Z c h R 7 5 e + e 4 O Y p g K l o Z x o 3 f W 9 x A Q Z U Y Z R j t N i L 9 W Y E D o m Q + x a K o l A H W T z R 6 f u u V U G b h Q r O 9 K 4 c / X 3 R U a E 1 h M R 2 k 1 B z E g v e z P x X y 8 U S 8 k m u g k y J p P U o K S L 4 C j l r o n d W Q v u g C m k h k 8 s I V Q x + 7 t L R 0 Q R a m x X R V u K v 1 z B K m l V K / 5 l p d q 4 K t d u 8 3 o K c A p n c A E + X E M N 7 q E O T a C A 8 A w v 8 O o 8 O W / O u / O x W F 1 z 8 p s T + A P n 8 w c D V p O U < / l a t e x i t > f < l a t e x i t s h a 1 _ b a s e 6 4 = " V G U b / B W S x t m + t g E O o T b 6 O S 8 F g e 4 = " > A A A B + H i c b V D L S g N B E O z 1 G e M r 6 t H L Y h A 8 h d 0 o 6 D H o x W M C 5 g H J E m Y n v c m Q m d l l Z l a I S 7 7 A q 9 6 9 i V f / x q t f 4 i T Z g y Y W N B R V 3 V R T Y c K Z N p 7 3 5 a y t b 2 x u b R d 2 i r t 7 + w e H p a P j l o 5 T R b F J Y x 6 r T k g 0 c i a x a Z j h 2 E k U E h F y b I f j u 5 n f f k S l W S w f z C T B Q J C h Z B G j x F i p E f V L Z a / i z e G u E j 8 n Z c h R 7 5 e + e 4 O Y p g K l o Z x o 3 f W 9 x A Q Z U Y Z R j t N i L 9 W Y E D o m Q + x a K o l A H W T z R 6 f u u V U G b h Q r O 9 K 4 c / X 3 R U a E 1 h M R 2 k 1 B z E g v e z P x X y 8 U S 8 k m u g k y J p P U o K S L 4 C j l r o n d W Q v u g C m k h k 8 s I V Q x + 7 t L R 0 Q R a m x X R V u K v 1 z B K m l V K / 5 l p d q 4 K t d u 8 3 o K c A p n c A E + X E M N 7 q E O T a C A 8 A w v 8 O o 8 O W / O u / O x W F 1 z 8 p s T + A P n 8 w c D V p O U < / l a t e x i t > N < l a t e x i t s h a 1 _ b a s e 6 4 = " P C r 4 p + G g r z i T k u L n C K z D 6 G V u y j Y = " > A A A B + H i c b V D L S g N B E O y N r x h f U Y 9 e F o O Q U 9 i N g h 4 D X j x J A u Y B y R J m J 7 3 J k J n Z Z W Z W i C F f 4 F X v 3 s S r f + P V L 3 G S 7 E E T C x q K q m 6 q q T D h T B v P + 3 J y G 5 t b 2 z v 5 3 c L e / s H h U f H 4 p K X j V F F s 0 p j H q h M S j Z x J b B p m O H Y S h U S E H N v h + H b u t x 9 R a R b L B z N J M B B k K F n E K D F W a t z 3 i y W v 4 i 3 g r h M / I y X I U O 8 X v 3 u D m K Y C p a G c a N 3 1 v c Q E U 6 I M o x x n h V 6 q M S F 0 T I b Y t V Q S g T q Y L h 6 d u R d W G b h R r O x I 4 y 7 U 3 x d T I r S e i N B u C m J G e t W b i / 9 6 o V h J N t F N M G U y S Q 1 K u g y O U u 6 a 2 J 2 3 4 A 6 Y Q m r 4 x B J C F b O / u 3 R E F K H G d l W w p f i r F a y T V r X i X 1 a q j a t S r Z z V k 4 c z O I c y + H A N N b i D O j S B A s I z v M C r 8 + S 8 O e / O x 3 I 1 5 2 Q 3 p / A H z u c P 1 Z W T Y g = = < / l a t e x i t > M < l a t e x i t s h a 1 _ b a s e 6 4 = " q o X z L / D J s b 8 q V J 7 v D 9 k d l + 8 R b P M = " > A A A B + H i c b V D L S g N B E O y N r x h f U Y 9 e F o O Q U 9 i N g h 4 D X r w I C Z g H J E u Y n f Q m Q 2 Z m l 5 l Z I Y Z 8 g V e 9 e x O v / o 1 X v 8 R J s g d N L G g o q r q p p s K E M 2 0 8 7 8 v J b W x u b e / k d w t 7 + w e H R 8 X j k 5 a O U 0 W x S W M e q 0 5 I N H I m s W m Y 4 d h J F B I R c m y H 4 9 u 5 3 3 5 E p V k s H 8 w k w U C Q o W Q R o 8 R Y q X H f L 5 a 8 i r e A u 0 7 8 j J Q g Q 7 1 f / O 4 N Y p o K l I Z y o n X X 9 x I T T I k y j H K c F X q p x o T Q M R l i 1 1 J J B O p g u n h 0 5 l 5 Y Z e B G s b I j j b t Q f 1 9 M i d B 6 I k K 7 K Y g Z 6 V V v L v 7 r h W I l 2 U Q 3 w Z T J J D U o 6 T I 4 S r l r Y n f e g j t g C q n h E 0 s I V c z + 7 t I R U Y Q a 2 1 X B l u K v V r B O W t W K f 1 m p N q 5 K t X J W T x 7 O 4 B z K 4 M M 1 1 O A O 6 t A E C g j P 8 A K v z p P z 5 r w 7 H 8 v V n J P d n M I f O J 8 / 1 A G T Y Q = = < / l a t e x i t > N ⇥ M < l a t e x i t s h a 1 _ b a s e 6 4 = " x Z m M f 1 9 O W c R q M J 0 g 7 P Q w 5 b o z Q N U = " > A A A C A H i c b V A 9 S w N B F H w X v 2 L 8 i l r a L A Y h V b i L g p Y B G x s l g k m U 5 A h 7 m 7 1 k y e 7 d s f t O C C G N f 8 F W e z u x 9 Z / Y + k v c J F d o 4 s C D Y e Y 9 5 j F B I o V B 1 / 1 y c i u r a + s b + c 3 C 1 v b O 7 l 5 x / 6 B p 4 l Q z 3 m C x j P V 9 Q A 2 X I u I N F C j 5 f a I 5 V Y H k r W B 4 O f V b j 1 w b E U d 3 O E q 4 r 2 g / E q F g F K 3 0 c N N B o b g h 1 9 1 i y a 2 4 M 5 B l 4 m W k B B n q 3 e J 3 p x e z V P E I m a T G t D 0 3 Q X 9 M N Q o m + a T Q S Q 1 P K B v S P m 9 b G l E b 4 4 9 n D 0 / I i V V 6 J I y 1 n Q j J T P 1 9 M a b K m J E K 7 K a i O D C L 3 l T 8 1 w v U Q j K G F / 5 Y R E m K P G L z 4 D C V B G M

i T k u L n C K z D 6 G V u y j Y = " > A A A B + H i c b V D L S g N B E O y N r x h f U Y 9 e F o O Q U 9 i N g h 4 D X j x J A u Y B y R J m J 7 3 J k J n Z Z W Z W i C F f 4 F X v 3 s S r f + P V L 3 G S 7 E E T C x q K q m 6 q q T D h T B v P + 3 J y G 5 t b 2 z v 5 3 c L e / s H h U f H 4 p K X j V F F s 0 p j H q h M S j Z x J b B p m O H Y S h U S E H N v h + H b u t x 9 R a R b L B z N J M B B k K F n E K D F W a t z 3 i y W v 4 i 3 g r h M / I y X I U O 8 X v 3 u D m K Y C p a G c a N 3 1 v c Q E U 6 I M o x x n h V 6 q M S F 0 T I b Y t V Q S g T q Y L h 6 d u R d W G b h R r O x I 4 y 7 U 3 x d T I r S e i N B u C m J G e t W b i / 9 6 o V h J N t F N M G U y S Q 1 K u g y O U u 6 a 2 J 2 3 4 A 6 Y Q m r 4 x B J C F b O / u 3 R E F K H G d l W w p f i r F a y T V r X i X 1 a q j a t S

Z Z R Q Z 6 f M C g X l Z o = " > A A A C L n i c b Z D L S g M x F I Y z X m u 9 j b p 0 M 1 g K L U i Z q Y q C m 4 I b l x X s B T q l Z N K 0 D U 0 m Q 3 K m W o Y + i S / h K 7 j V v e B C B F c + h u l l U V t / C P x 8 5 x z O y R 9 E n G l w 3 Q 9 r Z X V t f W M z t Z X e 3 t n d 2 7 c P D q t a x o r Q C p F c q n q A N e U s p B V g w G k 9 U h S L g N N a 0 L 8 Z 1 2 s D q j S T 4 T 0 M I 9 o U u B u y D i M Y D G r Z F 9 D y c n 4 g k s d R P p 3 1 N e s K n P N J W 8 L 1 m A 4 m V P X k H H s Y 5 V t 2 x i 2 4 E z n L x p u Z D J q p 3 L K / / b Y k s a A h E I 6 1 b n h u B M 0 E K 2 C E 0 1 H a j z W N M O n j L m 0 Y G 2 J B d T O Z f G / k Z A 1 p O x 2 p z A v B m d D 5 i Q Q L r Y c i M J 0 C Q 0 8 v 1 s b w 3 1 o g F j Z D 5 6 q Z s D C K g Y Z k u r g T c w e k M 8 7 O a T N F C f C h M Z g o Z m 5 3 S A 8 r T M A k n D

a h e I s R L J t q s e C d F Y p 3 5 5 n S 6 S y e F D p G J y i H P H S J S u g W l V E F E f S E X t A r e r O e r X f r 0 / q a t q 5 Y s 5 k j 9 E f W z y / O B K g I < / l a t e x i t > tM (x) < l a t e x i t s h a 1 _ b a s e 6 4 = " e y K E 6 B 4 b e j J r 9

E r f j R C 9 W W g Z n 2 8 = " > A A A C L n i c b Z D L S g M x F I Y z 9 V b r r e r S z W A p t C B l p i o K b g p u 3 A g V 7 A U 6 p W T S t A 1 N J k N y p l q G P o k v 4 S u 4 1 b 3 g Q g R X P o b p Z V F b f w j 8 f O c c z s n v h 5 x p c J w P K 7 G y u r a + k d x M b W 3 v 7 O 6 l 9 w + q W k a K 0 A q R X K q 6 j z X l L K A V Y M B p P V Q U C 5 / T m t + / H t d r A 6 o 0 k 8 E 9 D E P a F L g b s A 4 j G A x q p c + h d Z v z f B E / j v K p r K d Z V + C c R 9 o S r s Z 0 M K G q J + f Y w y j f S m e c g j O R v W z c m c m g m c q t 9 L f X l i Q S N A D C s d Y N 1 w m h G W M F j H A 6 S n m R p i E m f d y l D W M D L K h u x p P v j e y s I W 2 7 I 5 V 5 A d g T O j 8 R Y 6 H 1 U P i m U 2 D o 6 c X a G P 5 b 8 8 X C Z u h c N m M W h B H Q g E w X d y J u g 7 T H 2 d l t p i g B P j Q G E 8 X M 7 T b p Y Y U J m I R T J h R 3 M Y J l U y 0 W 3 N N C 8 e 4 s U z q Z x Z N E R + g Y 5 Z C L L l A J 3 a A y q i C C n t A L e k V v 1 r P 1 b n 1 a X 9 P W h D W b O U R / Z P 3 8 A v + s q C Q = < / l a t e x i t > Aggregation (Cat, 0 ) < l a t e x i t s h a 1 _ b a s e 6 4 = " v p 5 7 g K W B o 2 u S s z 5 0 R K u A D 2 Y 3 x 4 c = " > A A A D P X i c b Z L P a 9 s w F M d l 7 1 f n / U q 7 4 y 5 i I d C C C X Y 6 a H s r d I c d O 7 a 0 h S g L s i w n o v I P p O f R Y J w / b J f 9 D 7 v t t s s O G 2 P

X X a c 4 x v n 5 w P D 1 5 7 2 n r / S k I J N C g + d 9 t + x 7 9 x 8 8 f L T 3 2 H n y 9 N n z F 6 3 9 g y u d 5 o r x P k t l q m 4 C q r k U C e + D A M l v M s V p H E h + H d x e z P P X n 7 n S I k 0 + w j T j w 5

i O E x E J R s G g 0 b 7 1 4 Z A A v 4 P i g k L p e p 8 I E 4 o d O R 0 i e Q S k M C L g Y 5 E U V C k 6 L Q t Z G r K s n y 3 0 2 3 R s e N 3 s z s 4 a 5 Z 8 2 s n d S S 6 e z y 9 G d r d K z J S Y y T E G 7 m B D c W a t p 1 t 7 o b Y y O X E I a s / k W t 8 0 q u m K 2 X r 7 D o e J L B z M M n o T 1 c M y P E u M J k H L U a n t d r w q 8 L f x a t F E d l 6 P W N x K m L I 9 5 A k x S r Q e + l 8 H Q r A u C S V 4 6 J N c 8 o + y W j v n A y I T G X A + L 6 v Z L 3 D E k x F G q z J c A r u h q R 0 F j r a d x Y C p j C h O 9 m Z v D X b l B D t H p s B B J l g N P 2 M I o y i W G F M + f E g 6 F 4 g z k 1 A j K l D B 7 x W x C F W V g H p x j h u B v H n l b X P W 6 / n G 3 9 / 5 N + 9 y r x 7 G H X q H X 6 B D 5 6 A S d o 3 f o E v U R s 7 5 Y P 6

x f 1 m / 7 q / 3 T / m P / X Z T a V t 3 z E q 2 F / e 8 / f q 4 F 5 w = = < / l a t e x i t > (Cat, 90 )

< l a t e x i t s h a 1 _ b a s e 6 4 = " 4 F C m K k K l 9 O W 1 d u 5 4 k Y H N o 2 k 9 k u 0 = " > A A A D P n i c b Z J N a 9 t A E I Z X 6 l e q f j n t s R d R Y 0 h A G M k t J L k F 0 k O P K c R J w O u a 1 W o l L 1 m t x O 6 o 1 A j 5 j / X S 3 9 B b j 7 3 0 0 F J 6 7 b F r W V X 8 N S B 4 9 M 7 M v q P R h r n g G n z / m 2 X f u X v v / o O 9 h 8 6 j x 0 + e P u v s P 7 / U W a E o G 9 J M Z O o 6 J J o J L t k Q O A h 2 n S t G 0 l C w q / D m b J G / + s i U 5 p m 8 g F n O x i l J J I 8 5 J W C k y b 5 1 c Y C B f Y L y j E D l n f g f M O W K H j o 9 L F g M u D Q Q s o T L k i h F Z l U p K q P c N s y X / D Z L j N 4 0 e / O T l o L j F g d H D T q 9 V c v / j t 5 8 5 y D G Q U Q Z a M / F 2 O 2 t 1 b R n b / S 2 R o c e x q 3 Z Y s R t s 1 p d M V s v 3 + F Q 6 7 c O Z h l M R s 1 y z I v i y R R w N e l 0 / b 5 f h 7 s N Q Q N d 1 M T 5 p P M V R x k t U i a B C q L 1 K P B z G J t z g V P B K g c X m u W E 3 p C E j Q x K k j I 9 L u v f X 7 k 9 o 0 R u n C n z S H B r d b W j J K n W s z Q 0 l S m B q d 7 M L c R d u V E B 8 f G 4 5 D I v g E m 6 N I o L 4 U L m L u 6 S G 3 H F K I i Z A U I V N 7 O 6 d E o U o W B u n G O W E G x + 8 j Z c D v r B 6 / 7 g / Z v u q d + s Y w + 9 R K / Q A Q r Q E T p F 7 9 A 5 G i J q f b a + W z + t X / Y X + 4 f 9 2 / 6 z L L W t p u c F W g v 7 7 z 9 J H g Y q < / l a t e x i t > (Cat, 180 ) < l a t e x i t s h a 1 _ b a s e 6 4 = " 1 t b 5 q 6 U D 1 t o w k Z t 4 1 Z z y w 7 c d 4 c c = " > A A A D P 3 i c b Z J L a 9 t A E M d X 6 i t V X 0 5 7 7 G W p M S Q g j O Q U k t w C 6 a H H F O o k 4 H X N a r W S l 6 w e 7 I 5 K j Z A + W S / 9 C r 3 1 2 k s P L a X X 3 r q W V T 9 i D w j + + s 3 M / k e j D X I p N H j e N 8 u + c / f e / Q d 7 D 5 1 H j 5 8 8 f d b Z f 3 6 p s 0 I x P m S Z z N R 1 Q D W X I u V D E C D 5 d a 4 4 T Q L J r 4 K b 8 3 n + 6 i N X W m T p e 5 j l f J z Q O B W R Y B Q M m u x b w w M C / B O U 5 x Q q 1 z / x P h A m F D t 0 e k T y C E h p R M B j k Z Z U K T q r S l k Z s u q o F / p N F h v e N r v 1 6 V I t T 3 T r w X E r n d 6 6 5 3 9 H t 1 6 n p y t M Z J i B d j E h u F f v n n a z d 2 l 0 6 B K y N J u P u G 3 W 0 D W z z f I d D g 1 f O Z h l 8 D R s l 2 N e l I i n Q K p J p + v 1 v S b w t v B b 0 U V t X E w 6 X 0 m Y s S L h K T B J t R 7 5 X g 5 j c y 4 I J n n l k E L z n L I b G v O R k S l N u B 6 X z f + v c M + Q E E e Z M k 8 K u K H r H S V N t J 4 l g a l M K E z 1 7 d w c 7 s q N C o h O x q V I 8 w J 4 y h Z G U S E x Z H h + m X A o F G c g Z 0 Z Q p o S Z F b M p V Z S B u X K O W Y J / + 5 O 3 x e W g 7 x / 1 B + 9 e d 8 + 8 d h 1 7 6 C V 6 h Q 6 Q j 4 7 R G X q L L t A Q M e u z 9 d 3 6 a f 2 y v 9 g / 7 N / 2 n 0 W p b b U 9 L 9 B G 2 H / / A f q P B m Q = < / l a t e x i t > (Cat, 270 ) < l a t e x i t s h a 1 _ b a s e 6 4 = " x / Q 5 c k E z I / M D M l e F n N C Z 4 Y B c V L 8 = " > A A A D P 3 i c b V L L a t w w F J X d V + q + J u 2 y G 9 F h I A U z 2 N N C k l 0 g X X S Z Q i c J j K a D L M s e E f m B d B 0 y G M + X d d N f 6 K 7 b b r p o K d 1 2 V 4 3 H 9 b x

y Q X B 0 7 j 3 3 S F c K c i k 0 e N 4 3 y 7 5 z 9 9 7 9 B 3 s P n U e P n z x 9 1 t l / f q 6 z Q ance with respect to self-supervision for the primary classifier performing the original task. Thus, utilizing the self-supervised labels in this way could hurt the performance, e.g., in fully supervised settings. This inspires us to revisit and explore data augmentation methods with self-supervision.

j E + Z J n M 1 G V A N Z c i 5 U M Q I P l l r j h N A s k v g q v T R f 7 i m i s t s v Q j z H I + T m i c i k g w C o a a 7 F v D A w L 8 B s p T C p U 7 O P Q + E S Y U e + 3 0 i O Q R k N K A g M c i L a l S d F a V s j L M S j F f 4 n d Z b P h G 7 M 6 P W + Q f t b B t 7 v T W P f 8 7 u v N 1 9 n h F E x l m o F 1 M C O 5 t 1 L S 9 t 7 S r W 7 i E t G a L I + 6 a 1 e y a 2 W b 5 L Q 4 1 v z E n n o b N c M x G i X g K p J p 0 u l 7 f q w P v A r 8 B X d T E 2 a T z l Y Q Z K x K e A p N U 6 5 H v 5 T A 2 f U E w y S u H F J r n l F 3 R m I 8 M T G n C 9 b i s 3 7 / C P c O E O M q U W S n g m l 1 X l D T R e p Y E p j K h M N X b u Q V 5 W 2 5 U Q H Q 0 L k W a F 8 B T t j S K C o k h w 4 v P h E O h O A M 5 M 4 A y J c x Z M Z t S R R m Y L + e Y I f j b V 9 4 F 5 4 O + / 6 Y / + P C 2 e + I 1 4 9 h D L 9 E r d I B 8 d I h O 0 H t 0 h o a I W Z + t 7 9 Z P 6 5 f 9 x f 5 h / 7 b / L E t t q 9 G 8 Q B t h / / 0 H + p A G Z A = = < / l a t e x i t > Self-distillation (•; u) < l a t e x i t s h a 1 _ b a s e 6 4 = " A 2 s K X v I z 3 J i T z / 2 B p 4 2 u K N K G d Q U = " > A A A C D X i c b V D L S s N A F J 3 U V 6 2 v + N i 5 G S x C B S l J F R T c F N y 4 r G A f 0 I Q y m U z b o f M I M x O h h n 6 D v + B W 9 + 7 E r d / g 1 i 9 x 2 m a h 1 Q M X D u f c y 7 m c K G F U G 8 / 7 d A p L y y u r a 8 X 1 0 s b m 1 v a O u 7 v X 0 j J V m D S x Z F J 1 I q Q J o 4 I 0 D T W M d B J F E I 8 Y a U e j 6 6 n f v i d K U y n u z D g h I U c D Q f s U I 2 O l n n s Q a D r g q B L g W J q r I O J Z O j n p u W W v 6 s 0 A / x I / J 2 W Q o 9 F z v 4 J Y 4 p Q T Y T B D W n d 9 L z F h h p S h m J F J K U g 1 S R A e o Q H p W i o Q J z r M Z t 9 P 4 L F V Y t i X y o 4 w c K b + v M g Q 1 3 r M I 7 v J k R n q R W 8 q / u t F f C H Z 9 C / D j I o k N U T g e X A / Z d B I O K 0 G x l Q R b N j Y E o Q V t b 9 D P E Q K Y W M L L N l S / M U K / p J W r e q f V W u 3 5 + X 6 a V 5 P E R y C I 1 A B P r g A d X A D G q A J M H g A T + A Z v D i P z q v z 5 r z P V w t O f r M P f s H 5 + A b Y 1 5 u e < / l a t e x i t > ⇢(•; w) < l a t e x i t s h a 1 _ b a s e 6 4 = " T V W u w n y H K x H T t 3 J 8 U 3 j R U 8 X G x d Y = " > A A A C C 3 i c b V D L S s N A F J 3 U V 6 2 v W J d u B o t Q Q U p S B Q U 3 B T c u K 9 g H N K F M J p N 2 6 E w m z E z U E v o J / o J b 3 b s T t 3 6 E W 7 / E a Z u F t h 6 4 c D j n X s 7 l B A m j S j v O l 1 V Y W V 1 b 3 y h u l r a 2 d 3 b 3 7 P 1 y W 4 l U Y t L C g g n Z D Z A i j M a k p a l m p J t I g n j A S C c Y X U / 9 z j 2 R i o r 4 T o 8 T 4 n M 0 i G l E M d J G 6 t t l T w 5 F 1 c O h 0 F d e w L O H y U n f r j g 1 Z w a 4 T N y c V E C O Z t / + 9 k K B U 0 5 i j R l S q u c 6 i f Y z J D X F j E x K X q p I g v A I D U j P 0 B h x o v x s 9 v s E H h s l h J G Q Z m I N Z + r v i w x x p c Y 8 M J s c 6 a F a 9 K b i v 1 7 A F 5 J 1 d O l n N E 5 S T W I 8 D 4 5 S B r W A 0 2 J g S C X B m o 0 N Q V h S 8 z v E Q y Q R 1 q a + k i n F X a x g m b T r N f e s V r 8 9 r z R O 8 3 q K 4 B A c g S p w w Q V o g B v Q B C 2 A w S N 4 B i / g 1 X q y 3 q x 3 6 2 O + W r D y m w P w B 9 b n D 0 d K m s Q = < / l a t e x i t > f < l a t e x i t s h a 1 _ b a s e 6 4 = " V G U b / B W S x t m + t g E O o T b 6 O S 8 F g e 4 = " > A A A B + H i c b V D L S g N B E O z 1 G e M r 6 t H L Y h A 8 h d 0 o 6 D H o x W M C 5 g H J E m Y n v c m Q m d l l Z l a I S 7 7 A q 9 6 9 i V f / x q t f 4 i T Z g y Y W N B R V 3 V R T Y c K Z N p 7 3 5 a y t b 2 x u b R d 2 i r t 7 + w e H p a P j l o 5 T R b F J Y x 6 r T k g 0 c i a x a Z j h 2 E k U E h F y b I f j u 5 n f f k S l W S w f z C T B Q J C h Z B G j x F i p E f V L Z a / i z e G u E j 8 n Z c h R 7 5 e + e 4 O Y p g K l o Z x o 3 f W 9 x A Q Z U Y Z R j t N i L 9 W Y E D o m Q + x a K o l A H W T z R 6 f u u V U G b h Q r O 9 K 4 c / X 3 R U a E 1 h M R 2 k 1 B z E g v e z P x X y 8 U S 8 k m u g k y J p P U o K S L 4 C j l r o n d W Q v u g C m k h k 8 s I V Q x + 7 t L R 0 Q R a m x X R V u K v 1 z B K m l V K / 5 l p d q 4 K t d u 8 3 o K c A p n c A E + X E M N 7 q E O T a C A

Contribution. We primarily focus on fully supervised learning setups, i.e., assume not only the original/primary labels for all training samples, but also self-supervised labels for the augmented samples. Our main idea is simple and intuitive (see Figure 1 (a)): maintain a single joint classifier, instead of two separate classifiers typically used in the prior self-supervision literature. For example, if the original and self-supervised tasks are CIFAR10 (10 labels) and rotation (4 labels), respectively, we learn the joint probability distribution on all possible combinations of 40 labels. This approach assumes no relationship between the original and self-supervised labels, and consequently does not force any invariance to the transformations. Furthermore, since we assign different self-supervised labels for each transformation, it is possible to make a prediction by aggregating across all transformations at test time, as illustrated in Figure 1 (b) . This can provide an (implicit) ensemble effect using a single model. Finally, to speed up the evaluation process, we also propose a novel knowledge transfer technique, self-distillation, which transfers the knowledge of the aggregated prediction into the model itself.

In our experiments, we consider two types of transformations for self-supervised data augmentation, rotation (4 transformations) and color permutation (6 transformations), as illustrated in Figure 1(c) and Figure 1(d) , respectively. We also consider composed transformations using both rotation and color permutation, i.e., up to 24 transformations. To show wide applicability of our method, we consider various image benchmark datasets and classification scenarios including the few-shot and imbalanced classification tasks. In all tested settings, our simple method improves the classification accuracy significantly and consistently. As desired, the gain tends to be larger when using more augmentation (or self-supervised) labels. We highlight some of our experimental results as follows:

• We show that the proposed self-supervised data augmentation methods with aggregation (SDA+AG) and self-distillation (SDA+SD) can provide significant improvements in the standard fully supervised settings. For example, by using 4 self-supervised labels of rotation, SDA+AG (or SDA+SD) achieves 8.60% (or 5.24%) and 18.8% (or 15.3%) relative gains on the CIFAR100 and CUB200 datasets, respectively, compared to the baseline. 1The gain is increased up to 20.8% in CUB200 by using 12 composed transformations of the rotation and color permutation.

• We show that SDA+AG can improve the state-of-the-art methods, ProtoNet (Snell et al., 2017) and MetaOptNet (Lee et al., 2019) , for few-shot classification, e.g., MetaOptNet with SDA+AG achieves 7.05% relative gain on 5-way 5-shot tasks on the FC100 dataset. • We show that SDA+SD can improve the state-of-the-art methods, Class-Balanced (Cui et al., 2019) and LDAM (Cao et al., 2019) , for imbalanced classification, e.g., LDAM with SDA+SD achieves 8.30% relative gain on an imbalanced version of the CIFAR100 dataset.

We remark that rotation and color permutation are rarely used in the literature for improving fully supervised learning. We think our results are useful to guide many interesting future directions.

2. SELF-SUPERVISED DATA AUGMENTATION:

In this section, we provide the details of our self-supervised data augmentation techniques. We first discuss conventional techniques on data augmentation and self-supervision with their limitations in Section 2.1. Then, we propose our learning framework for data augmentation that can fully utilize the power of self-supervision. In Section 2.2, we also introduce a self-distillation technique transferring the augmented knowledge into the model itself for accelerating the inference speed.

Notation. Let x ∈ R d be an input, y ∈ {1, . . . , N } be its label where N is the number of classes, L CE be the cross-entropy loss function, σ(•; u) be the softmax classifier, i.e., σ i (z; u) = exp(u i z)/ k exp(u k z), and z = f (x; θ) be an embedding vector of x where f is a neural network with the parameter θ. We also let x = t(x) denote an augmented sample using a transformation t, and z = f ( x; θ) be the embedding of the augmented sample.

2.1. DATA AUGMENTATION AND SELF-SUPERVISION:

Data augmentation. In a supervised setting, the conventional data augmentation aims to improve upon the generalization ability of the target neural network f by leveraging certain transformations that can preserve their semantics, e.g. cropping, contrast enhancement, and flipping. We can write the training objective L DA with data augmentation as follows:

EQUATION

where T is a distribution of the transformations for data augmentation. Optimizing the above loss forces the classifier σ(f (•; θ); u) to be invariant to the transformations. However, depending on the type of transformations, forcing such invariance may not make sense, as the statistical characteristics of the augmented training samples could become very different from those of original training samples (e.g., rotation). In such a case, enforcing invariance to those transformations would make the learning more difficult, and even degrade the performance (see Table 1 in Section 3.2).

Self-supervision. The recent self-supervised learning literature (Zhang et al., 2019; Doersch et al., 2015; Noroozi & Favaro, 2016; Larsson et al., 2017; Oord et al., 2018; Gidaris et al., 2018) has shown that high-level semantic representations can be learned by predicting labels that could be obtained from the input signals without any human annotations. In self-supervised learning, models learn to predict which transformation t is applied to an input x given a modified sample x = t(x).

The common approach to utilize self-supervised labels is to optimize two losses of the original and self-supervised tasks, while sharing the feature space among them; that is, the two tasks are trained in a multi-task learning framework (Hendrycks et al., 2019; Zhai et al., 2019; Chen et al., 2019) . Thus, in a fully supervised setting, one can formulate the multi-task objective L MT with self-supervision as follows:

EQUATION

where {t j } M j=1 is a set of pre-defined transformations, M is the number of self-supervised labels, σ(•; v) is the classifier for self-supervision, and xj = t j (x). The above loss also forces the primary classifier σ(f (•; θ); u) to be invariant to the transformations {t j }. Therefore, due to the aforementioned reason, the usage of additional self-supervised labels does not guarantee the performance improvement, in particular, for fully supervised settings (see Table 1 in Section 3.2).

2.2. ELIMINATING INVARINCE VIA JOINT-LABEL CLASSIFIER:

Our key idea is to remove the unnecessary invariant property of the classifier σ(f (•; θ); u) in ( 1) and ( 2) among the transformed samples. To this end, we use a joint softmax classifier ρ(•; w) which represents the joint probability as P (i, j| x) = ρ ij ( z; w) = exp(w ij z)/ k,l exp(w kl z). Then, our training objective can be written as

EQUATION

where L CE (ρ( z; w), (i, j)) =log ρ ij ( z; w). Note that the above objective can be reduced to the data augmentation objective (1) when w ij = u i for all i, and the multi-task learning objective (2) when w ij = u i + v j for all i, j. It means that ( 3) is easier to optimize than (2) since both consider the same set of multi-labels, but latter forces the additional constraint w ij = u i +v j . The difference between the conventional augmentation, multi-task learning, and ours is illustrated in Figure 1(a) .

During training, we feed all M augmented samples simultaneously for each iteration as Gidaris et al. ( 2018) did, i.e., we minimize 1 |B| (x,y)∈B L SDA (x, y; θ, w) for each mini-batch B. We also assume the first transformation is the identity function, i.e., x1 = t 1 (x) = x.

Aggregated inference. Given a test sample x or its augmented sample xj = t j (x) by a transformation t j , we do not need to consider all N × M labels for the prediction of its original label, because we already know which transformation is applied. Therefore, we predict a label using the conditional probability P (i| xj , j) = exp(w ij zj )/ k exp(w kj zj ) where zj = f ( xj ; θ). Furthermore, for all possible transformations {t j }, we aggregate the corresponding conditional probabilities to improve the classification accuracy, i.e., we train a single model, which can perform inference as an ensemble model. To compute the probability of the aggregated inference, we first average pre-softmax activations, and then compute the softmax probability as follows:

P aggregated (i|x) = exp(s i ) N k=1 exp(s k )

where

s i = 1 M M j=1 w ij zj .

Since we assign different labels for each transformation t j , our aggregation scheme improves accuracy significantly. Somewhat surprisingly, it achieves comparable performance with the ensemble of multiple independent models in our experiments (see Table 2 in Section 3.2). We refer to the counterpart of the aggregation as single inference, which uses only the non-augmented or original sample x1 = x, i.e., predicts a label using P (i| x1 , 1).

Self-distillation from aggregation. Although the aforementioned aggregated inference achieves outstanding performance, it requires to compute zj = f ( xj ; θ) for all j, i.e., it requires M times higher computation cost than the single inference. To accelerate the inference, we perform selfdistillation (Hinton et al., 2015; Lan et al., 2018) from the aggregated knowledge P aggregated (•|x) to another classifier σ(f (x; θ); u) parameterized by u, as illustrated in Figure 1(b) . Then, the classifier σ(f (x; θ); u) can maintain the aggregated knowledge using only one embedding z = f (x; θ). To this end, we optimize the following objective:

L SDA+SD (x, y; θ, w, u) = L SDA (x, y; θ, w) +D KL (P aggregated (•|x) σ(f (x; θ); u)) + βL CE (σ(f (x; θ); u), y) (4)

where β is a hyperparameter and we simply choose β ∈ {0, 1}. When computing the gradient of L SDA+SD , we consider P aggregated (•|x) as a constant. After training, we use σ(f (x; θ); u) for inference without aggregation.

3. EXPERIMENTS:

We experimentally validate our self-supervised data augmentation techniques described in Section 2. Throughout this section, we refer to data augmentation (1) as DA, multi-task learning (2) as MT, and ours self-supervised data augmentation (3) as SDA for notational simplicity. After training with SDA, we consider two inference schemes: the single inference and the aggregated inference denoted by SDA+SI and SDA+AG, respectively. We also denote the self-distillation method (4) as SDA+SD which uses only the single inference σ(f (x; θ); u).

3.1. SETUP:

Datasets and models. We evaluate our method on various classification datasets: CIFAR10/100 (Krizhevsky et al., 2009) , Caltech-UCSD Birds or CUB200 (Wah et al., 2011) , Indoor Scene Recognition or MIT67 (Quattoni & Torralba, 2009) , Stanford Dogs (Khosla et al., 2011) , and tiny-ImageNet2 for standard or imbalanced image classification; mini-ImageNet (Vinyals et al., 2016) , CIFAR-FS (Bertinetto et al., 2019) , and FC100 (Oreshkin et al., 2018) for few-shot classification. Note that CUB200, MIT67, and Stanford Dogs are fine-grained datasets. We use residual networks (He et al., 2016) for all experiments: 32-layer ResNet for CIFAR, 18-layer ResNet for three finegrained datasets and tiny-ImageNet, and ResNet-12 (Lee et al., 2019) for few-shot datasets.

Implementation details. For the standard image classification datasets, we use SGD with learning rate of 0.1, momentum of 0.9, and weight decay of 10 -4 . We train for 80k iterations with batch size of 128. For the fine-grained datasets, we train for 30k iterations with batch size of 32 because they have a relatively smaller number of training samples. We decay the learning rate by the constant factor of 0.1 at 50% and 75% iterations. We report the average accuracy of 3 trials for all experiments. For few-shot learning and imbalance experiments, we use the publicly available codes of MetaOptNet (Lee et al., 2019) and LDAM (Cao et al., 2019) , respectively.

Choices of transformation. Since using the entire input image during training is important in achieving improved classification accuracy, some self-supervision techniques are not suitable for our purpose. For example, the Jigsaw puzzle approach (Noroozi & Favaro, 2016) divides an input image to 3×3 patches, and computes their embedding separately. Prediction using the embedding performs worse than that using the entire image. To avoid this issue, we choose two transformations which use the entire input image without cropping: rotation (Gidaris et al., 2018) and color permutation.

Rotation constructs M = 4 rotated images (0 • , 90 • , 180 • , 270 • ) as illustrated in Figure 1 (c). This transformation is widely used for self-supervision due to its simplicity, e.g., Chen et al. (2019) . Color permutation constructs M = 3! = 6 different images via swapping RGB channels as illustrated in Figure 1 (d) . This transformation can be useful when color information is important such as finegrained classification datasets. Comparison with DA and MT. We first verify that our proposed method can utilize self-supervision without loss of accuracy on fully supervised datasets while data augmentation and multi-task learning approaches cannot. To this end, we train models on generic classification datasets, CIFAR10/100 and tiny-ImageNet, using three different objectives: data augmentation L DA (1), multi-task learning L MT (2), and our self-supervised data augmentation L SDA (3) with rotation. As reported in Table 1 , L DA and L MT degrade the performance significantly compared to the baseline that does not use the rotation-based augmentation. However, when training with L SDA , the performance is improved. Figure 2 shows the classification accuracy on training and test samples of CIFAR100 while training. As shown in the figure, L DA causes a higher generalization error than others because L DA forces the unnecessary invariant property. Moreover, optimizing L MT is harder than doing L SDA as described in Section 2.2, thus the former achieves the lower accuracy on both training and test samples than the latter. These results show that learning invariance to some transformations, e.g., rotation, makes optimization harder and degrades the performance. Namely, such transformations should be carefully handled.

3.2. ABLATION STUDY:

Comparison with ten-crop and ensemble. Next, to evaluate the effect of aggregation in SDAtrained models, we compare the aggregation using rotation with other popular aggregation schemes: the ten-crop (Krizhevsky et al., 2012) method which aggregates the prediction scores over a number of cropped images, and the independent ensemble which aggregates the scores over four independently trained models. Note that the ensemble requires 4× more parameters than ours and ten-crop. Surprisingly, as reported in Table 2 , the aggregation using rotation performs significantly better than ten-crop, and also achieves competitive performance compared to the ensemble with 4 independently trained models. When using both independent ensemble and aggregation with rotation, i.e., the same number of parameters as the ensemble, the accuracy is improved further.

3.3. EVALUATION ON STANDARD SETTING:

Basic transformations. We demonstrate the effectiveness of our self-supervised augmentation method on various image classification datasets: CIFAR10/100, CUB200, MIT67, Stanford Dogs, and tiny-ImageNet. We first evaluate the effect of aggregated inference P aggregated (•|x) described in Section 2.2: see the SDA+AG column in Table 3 . Using rotation as augmentation improves the classification accuracy on all datasets, e.g., 8.60% and 18.8% relative gain over baselines on CIFAR100 and CUB200, respectively. With color permutation, the performance improvements are less significant on CIFAR and tiny-ImageNet, but it still provides meaningful gains on fine-grained datasets, e.g., 12.6% and 10.6% relative gain on CUB200 and Stanford Dogs, respectively.

To maintain the performance of the aggregated inference and reduce its computation cost simultaneously, we apply the self-distillation method described in Section 2.2. As reported in the SDA+SD column in Table 3 , it also significantly improves the performance of the single inference (without aggregation) up to 16.1% and 12.4% relatively based on rotation and color permutation, respectively.

Composed transformations. We now show one can improve the performance further by using various combinations of two transformations, rotation and color permutation, on CUB and Stanford Dogs datasets. To construct the combinations, we first choose two subsets T r and T c of rotation and color permutation, respectively, e.g., T r = {0 • , 180 • } or T c = {RGB, GBR, BRG}. Then, let T = T r × T c be a set of composed transformations of all t r ∈ T r and t c ∈ T c . It means that t = (t r , t c ) ∈ T rotates an image by t r and then swaps color channels by t c . We first evaluate how the set of transformations T affects training. As reported in Table 4 , using a larger set T achieves better performance on both the single and aggregated inference than a smaller set T ⊂ T in most cases. However, under too many transformations, the aggregation performance can be degraded since the optimization becomes too harder. When using M = 12 transformations, we achieve the best performance, 20.8% and 15.4% relatively higher than baselines on CUB200 and Stanford Dogs, respectively.

Next, we demonstrate the aggregation effect at test time depending on various combinations of transformations. To this end, we use the model trained with M = 12 transformations, and evaluate the aggregated performance using all possible 2 M combinations. Figure 3 shows the performance of the aggregated inference depending on the size of combinations. As shown in the figure, the average is consistently increasing as the number of transformations increases. We also observe that the maximum performance of M = 4 is similar to that of M = 12, e.g., the aggregation of T = {(0 • , GBR), (90 • , BRG), (180 • , RGB), (270 • , RGB)} achieves 65.1%, while that of all 12 transformations achieves 65.4% on CUB200. Namely, one can choose a proper subset of transformations for saving inference cost and maintaining the aggregation performance simultaneously.

3.4. EVALUATION ON LIMITED-DATA SETTING:

Limited-data regime. Our augmentation techniques are also effective when only few training samples are available. To evaluate the effectiveness, we first construct sub-datasets of CIFAR100 via randomly choosing n ∈ {25, 50, 100, 250} samples for each class, and then train models with and without our rotation-based self-supervised data augmentation. As shown in Figure 4 , our scheme improves the accuracy relatively up to 37.5% under aggregation and 21.9% without aggregation.

Few-shot classification. Motivated by the above results in the limited-data regime, we also apply our SDA+AG method to solve few-shot classification, combined with the state-of-the-art methods, ProtoNet (Snell et al., 2017) and MetaOptNet (Lee et al., 2019) specialized for this problem. Note that our method augments N -way K-shot tasks to N M -way K-shot when using M -way transfor- mations. As reported in Table 5 , ours improves consistently 5-way 1/5-shot classification accuracy on mini-ImageNet, CIFAR-FS, and FC100. For example, we obtain 7.05% relative improvements on 5-shot tasks of FC100.

Imbalanced classification. Finally, we consider a setting where training datasets are imbalanced, where the number of instances per class largely differs and some classes have only a few training instances. For this experiment, we combine our SDA+SD method with two recent approaches, Class-Balanced (CB) loss (Cui et al., 2019) and LDAM (Cao et al., 2019) , specialized for this problem. Under imbalanced datasets of CIFAR10/100 which have long-tailed label distributions, our approach consistently improves the classification accuracy as reported in Table 6 (13.3% relative gain on imbalanced CIFAR100 datasets). These results show that our self-supervised data augmentation can be useful for various scenarios of limited training data.

4. CONCLUSION:

In this paper, we proposed a simple yet effective data augmentation approach, where we introduce self-supervised learning tasks as auxiliary tasks and train the model to jointly predict both the label for the original problem and the type of transformation, and validated it with extensive experiments on diverse datasets. We believe that our work could bring in many interesting directions for future research; for instance, one can revisit prior works on applications of self-supervision, e.g., training generative adversarial networks with self-supervision (Chen et al., 2019) . Applying our joint learning framework to fully-supervised tasks other than the few-shot or imbalanced classification task, or learning to select tasks that are helpful toward improving the main task prediction accuracy, are other interesting research directions we could further explore.

:

In this case, SDA+SD is 4 times faster than SDA+AG, as the latter aggregates predictions of 4 rotations.

https://tiny-imagenet.herokuapp.com/

Figures and Tables:

< l a t e x i t s h a 1 _ b a s e 6 4 = " V G U b / B W S x t m + t g E O o T b 6 O S 8 F g e 4 = " >A A A B + H i c b V D L S g N B E O z 1 G e M r 6 t H L Y h A 8 h d 0 o 6 D H o x W M C 5 g H J E m Y n v c m Q m d l l Z l a I S 7 7 A q 9 6 9 i V f / x q t f 4 i T Z g y Y W N B R V 3 V R T Y c K Z N p 7 3 5 a y t b 2 x u b R d 2 i r t 7 + w e H p a P j l o 5 T R b F J Y x 6 r T k g 0 c i a x a Z j h 2 E k U E h F y b I f j u 5 n f f k S l W S w f z C T B Q J C h Z B G j x F i p E f V L Z a / i z e G u E j 8 n Z c h R 7 5 e + e 4 O Y p g K l o Z x o 3 f W 9 x A Q Z U Y Z R j t N i L 9 W Y E D o m Q + x a K o l A H W T z R 6 f u u V U G b h Q r O 9 K 4 c / X 3 R U a E 1 h M R 2 k 1 B z E g v e z P x X y 8 U S 8 k m u g k y J p P U o K S L 4 C j l r o n d W Q v u g C m k h k 8 s I V Q x + 7 t L R 0 Q R a m x X R V u K v 1 z B K m l V K / 5 l p d q 4 K t d u 8 3 o K c A p n c A E + X E M N 7 q E O T a C A 8 A w v 8 O o 8 O W / O u / O x W F1 z 8 p s T + A P n 8 w c D V p O U < / l a t e x i t > f < l a t e x i t s h a 1 _ b a s e 6 4 = " V G U b / B W S x t m + t g E O o T b 6 O S 8 F g e 4 = " > A A A B + H i c b V D L S g N B E O z 1 G e M r 6 t H L Y h A 8 h d 0 o 6 D H o x W M C 5 g H J E m Y n v c m Q m d l l Z l a I S 7 7 A q 9 6 9 i V f / x q t f 4 i T Z g y Y W N B R V 3 V R T Y c K Z N p 7 3 5 a y t b 2 x u b R d 2 i r t 7 + w e H p a P j l o 5 T R b F J Y x 6 r T k g 0

y b Y P 0 h O Y M 5 c g S y r S w v x M 2 o J o y t J 0 V b C n e Y g X L p F m t e K e V 6 u 1 Z q V b O 6 s n D E R x D G T w 4 h x p c Q R 0 a w E D B M 7 z A q / P k v D n v z s d 8 N e d k N 4 f w B 8 7 n D 6 Z F l p 0 = < / l a t e x i t > N < l a t e x i t s h a 1 _ b a s e 6 4 = " P C r 4 p + G g r z

r Z z V k 4 c z O I c y + H A N N b i D O j S B A s I z v M C r 8 + S 8 O e / O x 3 I 1 5 2 Q 3 p / A H z u c P 1 Z W T Y g = = < / l a t e x i t > (a) Difference with previous approaches.

t e x i t s h a 1 _ b a s e 6 4 = " 9 S V b 3 c p z l J 4 b q

8 A w v 8 O o 8 O W / O u / O x W F 1 z 8 p s T + A P n 8 w c D V p O U < / l a t e x i t > (b) Aggregation & self-distillation. (c) Rotation (M = 4). (d) Color permutation (M = 6).

Figure 1: (a) An overview of our self-supervised data augmentation and previous approaches with self-supervision. (b) Illustrations of our aggregation method utilizing all augmented samples and self-distillation method transferring the aggregated knowledge into itself. (c) Rotation-based augmentation. (d) Color-permutation-based augmentation.

Figure 2: Training/test accuracy of DA, MT, SDA while training on CIFAR100.

Figure 3: Relative improvements of aggregation versus the number of transformations.

Figure 4: Relative improvements over baselines on subsets of CIFAR100.

Classification accuracy (%) of single inference using data augmentation (DA), multi-task learning (MT), and ours self-supervised data augmentation (SDA) with rotation. The best accuracy is indicated as bold, and the relative gain over the baseline is shown in brackets.

Table content: <table><tr><td>Dataset</td><td>Baseline</td><td>DA</td><td>MT</td><td>SDA+SI</td></tr><tr><td>CIFAR10</td><td>92.39</td><td colspan="3">90.44 (-2.11%) 90.79 (-1.73%) 92.50 (+0.12%)</td></tr><tr><td>CIFAR100</td><td>68.27</td><td colspan="3">65.73 (-3.72%) 66.10 (-3.18%) 68.68 (+0.60%)</td></tr><tr><td>tiny-ImageNet</td><td>63.11</td><td colspan="3">60.21 (-4.60%) 58.04 (-8.03%) 63.99 (+1.39%)</td></tr></table>

Classification accuracy (%) of the ten-crop, indenpendent ensemble, and our aggregation using rotation (SDA+AG). The best accuracy is indicated as bold, and the relative gain over the baseline is shown in brackets.

Table content: <table><tr><td/><td/><td colspan="2">Single Model</td><td/><td>4 Models</td></tr><tr><td>Dataset</td><td colspan="2">Baseline ten-crop</td><td>SDA+AG</td><td colspan="2">Ensemble Enemble + SDA+AG</td></tr><tr><td>CIFAR10</td><td>92.39</td><td>93.33</td><td>94.50 (+2.28%)</td><td>94.36</td><td>95.10 (+2.93%)</td></tr><tr><td>CIFAR100</td><td>68.27</td><td>70.54</td><td>74.14 (+8.60%)</td><td>74.82</td><td>76.40 (+11.9%)</td></tr><tr><td>tiny-ImageNet</td><td>63.11</td><td>64.95</td><td>66.95 (+6.08%)</td><td>68.18</td><td>69.01 (+9.35%)</td></tr></table>

Classification accuracy (%) using self-supervised data augmentation with rotation and color permutation. SDA+SD and SDA+AG indicate the single inference trained by L SDA+SD , and the aggregated inference trained by L SDA , respectively. The relative gain is shown in brackets.

Table content: <table><tr><td/><td/><td colspan="2">Rotation</td><td colspan="2">Color Permutation</td></tr><tr><td>Dataset</td><td>Baseline</td><td>SDA+SD</td><td>SDA+AG</td><td>SDA+SD</td><td>SDA+AG</td></tr><tr><td>CIFAR10</td><td>92.39</td><td colspan="4">93.26 (+0.94%) 94.50 (+2.28%) 91.51 (-0.95%) 92.51 (+0.13%)</td></tr><tr><td>CIFAR100</td><td>68.27</td><td colspan="4">71.85 (+5.24%) 74.14 (+8.60%) 68.33 (+0.09%) 69.14 (+1.27%)</td></tr><tr><td>CUB200</td><td>54.24</td><td colspan="4">62.54 (+15.3%) 64.41 (+18.8%) 60.95 (+12.4%) 61.10 (+12.6%)</td></tr><tr><td>MIT67</td><td>54.75</td><td colspan="4">63.54 (+16.1%) 64.85 (+18.4%) 60.03 (+9.64%) 59.99 (+9.57%)</td></tr><tr><td>Stanford Dogs</td><td>60.62</td><td colspan="4">66.55 (+9.78%) 68.70 (+13.3%) 65.92 (+8.74%) 67.03 (+10.6%)</td></tr><tr><td>tiny-ImageNet</td><td>63.11</td><td colspan="4">65.53 (+3.83%) 66.95 (+6.08%) 63.98 (+1.38%) 64.15 (+1.65%)</td></tr></table>

Classification accuracy (%) depending on the set of transformations. The best accuracy is indicated as bold.

Table content: <table><tr><td>CUB200</td><td>Stanford Dogs</td></tr></table>

Average classification accuracy (%) with 95% confidence intervals of 1000 5-way few-shot tasks on mini-ImageNet, CIFAR-FS, and FC100. † and ‡ indicates 4-layer convolutional and 28layer residual networks(Zagoruyko & Komodakis, 2016), respectively. Others use 12-layer residual networks asLee et al. (2019). The best accuracy is indicated as bold.Lee et al., 2019) 62.64±0.61 78.63±0.46 72.0±0.7 84.2±0.5 41.1±0.6 55.5±0.6 ProtoNet (Snell et al., 2017) 59.25±0.64 75.60±0.48 72.2±0.7 83.5±0.5 37.5±0.6 52.5±0.6 ProtoNet + SDA+AG (ours) 62.22±0.69 77.78±0.51 74.6±0.7 86.8±0.5 40.0±0.6 55.7±0.6 MetaOptNet-RR (Lee et al., 2019) 61.41±0.61 77.88±0.46 72.6±0.7 84.3±0.5 40.5±0.6 55.3±0.6 MetaOptNet-RR + SDA+AG (ours) 62.93±0.63 79.63±0.47 73.5±0.7 86.7±0.5 42.2±0.6 59.2±0.5

Table content: <table><tr><td/><td colspan="2">mini-ImageNet</td><td colspan="2">CIFAR-FS</td><td colspan="2">FC100</td></tr><tr><td>Method</td><td>1-shot</td><td>5-shot</td><td>1-shot</td><td>5-shot</td><td>1-shot</td><td>5-shot</td></tr><tr><td>MAML  † (Finn et al., 2017) R2D2  † (Bertinetto et al., 2019) RelationNet  † (Sung et al., 2018)</td><td colspan="4">48.70±1.84 63.11±0.92 58.9±1.9 71.5±1.0 --65.3±0.2 79.4±0.1 50.44±0.82 65.32±0.70 55.0±1.0 69.3±0.8</td><td>---</td><td>---</td></tr><tr><td>SNAIL (Mishra et al., 2018)</td><td colspan="2">55.71±0.99 68.88±0.92</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td>TADAM (Oreshkin et al., 2018)</td><td colspan="2">58.50±0.30 76.70±0.30</td><td>-</td><td>-</td><td colspan="2">40.1±0.4 56.1±0.4</td></tr><tr><td>LEO  ‡ (Rusu et al., 2019)</td><td colspan="2">61.76±0.08 77.59±0.12</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td>MetaOptNet-SVM (</td><td/><td/><td/><td/><td/><td/></tr></table>

Classification accuracy (%) on imbalance datasets of CIFAR10/100. Imbalance Ratio is the ratio between the numbers of samples of most and least frequent classes. The best accuracy is indicated as bold, and the relative gain is shown in brackets. SDA+SD (ours) 80.24 (+4.17%) 89.58 (+1.61%) 45.53 (+8.30%) 59.89 (+1.67%)

Table content: <table><tr><td/><td colspan="2">Imbalanced CIFAR10</td><td colspan="2">Imbalanced CIFAR100</td></tr><tr><td>Imbalance Ratio (N max /N min )</td><td>100</td><td>10</td><td>100</td><td>10</td></tr><tr><td>Baseline</td><td>70.36</td><td>86.39</td><td>38.32</td><td>55.70</td></tr><tr><td>Baseline + SDA+SD (ours)</td><td colspan="4">74.61 (+6.04%) 89.55 (+3.66%) 43.42 (+13.3%) 60.79 (+9.14%)</td></tr><tr><td>CB-RW (Cui et al., 2019)</td><td>72.37</td><td>86.54</td><td>33.99</td><td>57.12</td></tr><tr><td>CB-RW + SDA+SD (ours)</td><td colspan="4">77.02 (+6.43%) 89.50 (+3.42%) 37.50 (+10.3%) 61.00 (+6.79%)</td></tr><tr><td>LDAM-DRW (Cao et al., 2019)</td><td>77.03</td><td>88.16</td><td>42.04</td><td>58.71</td></tr><tr><td>LDAM-DRW +</td><td/><td/><td/><td/></tr></table>

Peer review (iclr2020):
This paper proposes to use data-augmentation as self-supervision, applied as a regularizer in fully supervised classification settings in order to improve the generalization accuracy of the classifier. It uses a joint (label x augmentation) prediction space, rather than a factorized space of labels and augmentations in distinct prediction branches (multi-task setting). The author argues that this alleviates the requirement of invariance w.r.t. the transformation for the primary classifier and allows to use more diverse augmentations. The author tests this hypothesis on various classification datasets (including few-shot and fine-grained), and also evaluates the approach with test-time augmentation - proposing a knowledge distillation method to alleviate its cost. The author show improved accuracies in these settings w.r.t. simple data-augmentation (DA) or multi-task (MT) baselines. I am unsure about the justification of the method’s advantage w.r.t. the baseline and MT. I have the impression that the performance improvements w.r.t. MT could be explained by the added capacity. Barring futher experimental validation that this is not the case, I find the contributions of the paper to be minor. I will detail these points hereunder. * Method * 1. The features learned by MT are not invariant by the transformations (otherwise the auxiliary classifier could not predict the transformation). The primary classifier in MT could behave in a similar way as the SDA method by aggregating features from a joint (label x augmentation) space, acting as a projection on the label subspace. Therefore, I do not see in MT the core limitation that SDA claims to address (enforcing extraneous invariance of the classifier). 2. I see the joint classifier in SDA as the bulk of the novelty: the test-time augmentation experiments and teacher/student models to relieve the costs of these test-time augmentations are good to have but do not seem novel, and their applicability is not restricted to the SDA approach. * Experiments * 3. It is my understanding that SDA uses FxN(M-1) more parameters in the classifier than the baseline and than MT (F: number of features). This provides more parameters for the network to absorb the data augmentations, and to perform better for the primary classification task. This added capacity could explain the accuracy gains with respect to MT and to the baseline. For a fair comparison, a similar capacity should be added to the baseline, to DA and to MT. For the baseline, this could be done e.g. by using a large F x N x M classifier, and using a pooling over the M dimension. For DA, this could be done by using a large F x N x M classifier, and selecting the FxN slice that corresponds to the augmentation used (just as was done for SDA, but without the data-augmentation loss term in eq. 2). For MT, this could be done by adding an additional fully-connected layer before the two heads. * Minor * typo Table 2 caption “indenpendent”