TransBoost: Improving the Best ImageNet Performance using Deep Transduction
Authors: Omer Belhasin
Guy Bar-Shalom
Ran El-Yaniv
Venue: NeurIPS 2022 Accept (2022)
Keywords: Deep Transductive Learning
Image Classification

Abstract: This paper deals with deep transductive learning, and proposes TransBoost as a procedure for fine-tuning any deep neural model to improve its performance on any (unlabeled) test set provided at training time. TransBoost is inspired by a large margin principle and is efficient and simple to use. Our method significantly improves the ImageNet classification performance on a wide range of architectures, such as ResNets, MobileNetV3-L, EfficientNetB0, ViT-S, and ConvNext-T, leading to state-of-the-art transductive performance.
Additionally we show that TransBoost is effective on a wide variety of image classification datasets. The implementation of TransBoost is provided at: https://github.com/omerb01/TransBoost .

TransBoost: Improving the Best ImageNet Performance using Deep Transduction
Omer Belhasin"
Guy Bar-Shalom"
Department of Computer Science Technion Israel Institute of Technology omer. be@cs. Rechnion.ac..3
Department of Computer Science
Technion Israel Institute of Technology guy bêcs technion. ac.
Ran El-Yaniv
Department of Computer Science
Technion Israel Institute of Technology, Deci.Al ran Pani@cs. technion on. ac il
Abstract
This paper deals with deep transductive learning, and proposes TransBoost as a procedure for fine-tuning any deep neural model to improve its performance on any (unlabeled) test set provided at training time. TransBoost is inspired by a large margin principle and is efficient and simple to use. Our method significantly improves the ImageNet classification performance on a wide range of architectures, such as ResNets, MobileNerV3-LL EfficientNe:BO. VIT-S, and ConvNext-T, leading to state-of-the transductive performance. Additionally we show that TransBoost is effective wide variety of image classification datasets. The implementation TronsBoost is provided at: https: / tiut........ com/ http///errbbroeeee htap/omerb0o//raaBooss TransBostt
Introduction
80.64
ImageNet Top-1 Acc. (%)
CNNs
Mobile-CNNs
82.46
10:11
83.67
+2.28
82.16
80
79.03
79.86
12.49
78.33
+2.31
Transfommers
81.15
82.05
81.39
81.38
+0078
76.15
80.38
+3.41
78.6
73.36
78
73.29
77.7
76.81
+1.03
70
75.78
69.76
Efficientietto
MobileNetV34
Strikes Back
ResNer50
Comvllet-r
VFF-
Swin-T
ResNet18
ResNet34
Resterr0
ResNet101
ReoNet151 552
Figure I: ImageNet top-1 accuracy gains of TransBoost (green) using representatives of CNNs (blue), Mobile- CNNs (yellow) ) Transformers (red), in comparison to standard inductive (fully supervised) performance.
Alternative learning frameworks that make use unlabeled data have gained considerable attention in recent years. These include reni-supervised learning and transductive learning, both of which were extensively studied in classical and statistical machine learning [ 2 before the advent of deep learning. In this article focus on deep transductive classification. In this setting, both labeled training sample and an (unlabeled) test sample are provided at training time. The goal is to predict only the labels of the given test instances accurately possible. In contrast, in standard
Equaal contribution.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

inductive classification, the goal is to train general model capable of predicting the labels of unseen test instances.
In semi-supervised learning (SSI.) we are also given a set of unlabeled examples, which is used for selecting model. Despite commonly held misconception, however, the goal of SSL remains inductive rather than transductive, and the trained model must predict labels for new instances (i.e., the unlabeled instances are used for training only).
Transductive models deliver higher accuracy gains compared with traditional inductive models, as we demonstrate in this paper in the context of image classification. Much of the power of transductive learning is achieved through analyzing the given test set as a group (see Section (3.4). Transductive prediction is thus especially useful whenever we can accumulate a test set of instances and then train a specialized model to predict their labels. While there are many possible relevant use cases, one that stands out is medical diagnosis. Here, daily weekly medical records can be accumulated and sent to the transductive predictor as a set. In Appendix A, we highlight a number of additional use cases.
Perhaps the best-known transductive classification method in classical machine learning is the transductive support vector machine (TSVM) [4, 5 While a support vector machine (SVM) [6] seeks to achieve general decision function, a TSVM attempts to reduce misclassifications of just the target test instances. To the best of our knowledge, in deep learning, transductive settings have only been addressed in the case of few-shot learning (t-FSL).
In this paper we address deep transductive learning and introduce TransBoost, a novel procedure that takes pretrained neural classification model, along with a (labeled) training set, and an (unlabeled) test set. The procedure fine-tunes the model to improve its performance on this particular test set. The TransBoost optimization procedure is inspired by TSVM and its objective attempts to approximate large margin principle. TransBoost implemented through a simple transductive loss component, which is combined with the standard cross-entropy loss.
We examine the effectiveness TransBoost by using a variety of pretrained neural networks, datasets, and baseline models. Our extensive study examines spectrum of ResNet architectures and variety of modern architectures such as ConvNext (7]. MobileNetv3 [8] and ViT [9]. Figure I highhiggts some of TransBoost's 's results on the ImageNet dataset, which demonstrate impressive and consistent accuracy gains, leading to state-of-thh-aar performance compared to standard (inductive) classification as well as versus SSL and t-FSL methods (see full results in Section (5.).
To summarize, the contributions of this paper are: (1) A novel deep transductive loss function, inspired by a large margin principle, which optimizes performance on a specific test set. (2) A simple and efficient transductive procedure for fine-tuning a pretrained neural model in order boost its performance over an inductive setting. (3) A comprehensive empirical study of the proposed fine-tuning procedure showing great benefits across a large number of architectures, datasets and baseline techniques.
2 Problem Formulation
As transductive learning has not been extensively researched the deep learning community, we begin by defining the transductive learning problem within the context of deep learning classification. We follow Vapnik's 's standard formulation from statistical learning theory [4]. Let P(X, Y) be a probability distribution over X' y, where X represents an input space (c.g.. raw image data), and y represents a label set corresponding to C' classes. The learner is provided with a set of labeled instances, S A ((11,1)).... (LL, WL)}, where x, E X, y/s E V, and a finite set of unlabeled instances, X,, & (I1+1..... "ittl...,,,.tt) IL+U), where x; E X. The objective is to label the unlabeled instances based on this data. Given deep neural network fo X' - Y, where 0 denotes its parameters, and loss function É: y x y -> R, we follow I (Setting 2) and define the (true) risk over the unlabeled set,
1 U U Σ
(x(())
(1)
Thus, we are given a labeled training set S 2 (Xi, Yi) selected i.i.d. according to P(X, Y), as well as a pretrained classification model fe. and we presume it was trained with the labeled set St- An independent test set S, & (X., Yu), of U samples is randomly and independently selected in
2

---

the same manner, and we are required to revise the given model based on both St and X so as to minimize Equation TT) without knowing the labels y,-
3 Related Work
Transductive learning (or transduction) was first formulated by Vapnik 4 10) who also introduced the first transductive algorithm - TSVM 5 which learns large margin hyperplane classifier from labeled training data while simultaneously forcing it to take into account the (unlabeled) test data. Whereas transduction received considerable attention in the context of classical machine learning it has only been briefly addressed in the context of deep neural networks.
122 3],
Recall that in (deep) transductive classification we are provided with both a labeled training sample and an unlabeled test sample. A related but different setting is semi-supervised learning (SSL), where in addition to the labeled training set, we are also given (typically large) unlabeled training sample. The objective is inductive namely, to guess the label of any unseen test sample (that was drawn from the same distribution). A simple observation that any SSL algorithm can be applied in transduction by using the given (unlabeled) test sample the unlabeled SSL training sample. Thus, algorithms can be used to solve transductive learning. This fact may explain a common inductive SSL algorithm does not need to use (and cannot rely on) the fact that it will only be asked about the given test points. In contrast, a (meaningful) transductive algorithm must rely on this knowledge. in Section 5.2 we demonstrate that our proposed transductive procedure substantially outperforms the SimCLR [1]] and SimCLRy2 [12 algorithms, where SimCLRy2 currently achieves sate-of-theear
such as I 12 13
confusion between SSL and transduction. Nevertheless,
contend in this paper. In fact,
SSL performance according to [4].
Recently, there has been flurry of research concerning transductive few-shot learning (t-FSL) [6 17 In t-FSL we are given a classifier for a certain classification task, such as cats versus dogs, as well as training set (called the support set) for unseen classes (referred to as ways), such as elephants, tigers and lizards, containing only few labeled samples per unseen class. A test set (called the query set), containing instances from the unseen classes, is given and the goal is to learn the additional classes so as to correctly classify these test instances. Clearly, this setting is transductive; however, the goal here is to learn from few labeled examples (c.g., one-shot, five-shot). Formally, a L-FSL. algorithm can be applied to transductive classification (by treating the training set as a support set). For the most part, however, methods for t-FSL cannot be effectively applied to large (non few-shot) training sets (e.g.. iLPC [7].. Nevertheless, in order to conduct a complete study in transductive classification, we chose two t-FSL representative algorithms as baselines for our experiments (see Section 5). Specifically, we consider the entropy minimization (Ent-min) method of [1], and the TIM-GD method of [16). With Ent-min, the Shannon entropy reduced by fine-tuning, whereas TIM-GD optimizes the mutual information of test instances using only the classifier weights with fine-tuning. Ent-min is a common t-FSL baseline while TIM-GD is a top performing 5-shot (-FSL algorithm according to the few-shot leaderboard [18).
4 TransBoost: Fine- Tone-Tuning via Transductive Learning
We now present TransBoost, a procedure that takes a pretrained neural model along with its (labeled) training set as well as an (unlabeled) test set. The procedure then fine-tunes the model to improve its performance for this specific test set. The TransBoost optimization objective, L, is driven by Equation
3 and approximated by Algorithm[] T which includes a novel transductive loss component in addition to the standard tross-entropy loss. Our transductive component, LTransBooat is introduced in Equation
3 as regularization term and is defined in Equation (2). LrransBoost is inspired by TSVMs [4] and follows a large margin principle.
Let us elaborate on our loss function, LrranBooost 2 which is applied on the set of test samples, X This loss function includes an unsupervised non- non-negative symmetric pairwise similarity function. S: X' x X + R. which measures the similarity of two input instances. In our implementation, we simply used an Lbabd score function applied on prediction (softmax) vectors generated by Jo: see Equation (4). S is only applied on pairs of test instances that are likely to belong to different classes. The likelihood is determined by using an implementation of the Kronecker delta function,
6 X X + {0, 1), which obtains I iff the instances are predicted to belong to different classes;
3

---

Algorithm 1: TransBoost Procedure
Input: Labeled train sample St (Xr, Yi). unlabeled test sample X.. model fa that was pretrained on
St. a supervised pointwise loss function E, implementations of TransBoost's ' functions; S, 8 and s. Hyperparameters: Number of epochs E, labeled batch size L', unlabeled batch size U', regularization parameter
for epoch + to E do
// Sample labeled and unlabeled batches in cyclical manner for S/ 2 (Xi, Y?) C St, X C X,, v. max[] [ + [+][0t
do
// Prepare random test sample pairs x, " + generate a random permutation X // Approximate the TransBoost loss
Liunsooa f-
for i +- to U" do TraanSooa f- LriasBoost + =(x)))((a (8(1) )S
end for
TransBooot f LicansRossa E 8(2,, if Σ (101, of 0 else 0
11
// Apply optimization step
0 - optimize [ E'= {(o(wz)) v.) + A LTannReooa
13
14
end for
15 end
Equation (5) for our implementation. Additionally, for each pair intensify the loss using the model's confidence in its predictions. We define x X' - R+ to be a confidence function, which for each instance, æ, gives class prediction confidence; see Equation (6) for our implementation. Finally, our transductive loss function, TransBoost, is
TransBooot X XulSe. 5,
Σ
Us
ISKjjUU
where Us Eiskjje 8(1;, æz). The final loss function of our fine-tuning procedure includes the transductive loss term (2 as a regularization term as follows,
Σ ((o(ss) +A LraansBond (X,, S, T.
1
labeledVinductive (standard) loss
unlabelad/rrnncchcrive
where is standard (inductive) pointwise loss, e.g., the cross-entropy loss and À E R is regulariza-
tion hyperparameter.
Using the optimization objective in Equation (3), we incentivize test sample pairs that are likely to be different in their classes to also be different their empirical class probabilities while preserving the prior knowledge of fe. Accordingly, given two test samples, (æ,, xz), which are predicted to be different in their classes by 8, we obtain higher cost (by S) if x; close (similar) xj in its prediction vector than if x; is far (dissimilar) from Zj. depending on the model's confidence in its predictions utilizing K.
The computation of the LitanBoosst loss component 2 is quadratic in U namely, its time complexity O((5 (())). ). For large test sets (e.g., ImageNet), this is prohibitively large. Therefore, we propose to approximate Lrransooust by using random subsets of pairs from the test This random sampling is applied for each minibatch. The pseudo-code of the optimization procedure is presented in Algorithm []] As can be seen, the size of the set of sampled test pairs used to approximate (2) is taken to be the batch size (U"), which allows for very fast computation of LransBoost but could be sub-optimal (this size not optimized).
4.1 TransBoost Implementation
While S, and Equation 2 can be implemented in many ways, we now describe our proposed implementation that was used to obtain all the results described in Section As can be seen below,

---

we attempted to instantiate Equation (2) in the simplest possible manner. The consideration of more sophisticated methods is left for future work (and an alternative choice is discussed Section (.5).
The unsupervised symmetric pairwise similarity function Sj was straightforwardly implemented using the L2 norm follows,
(z)) A /2 - (o) - -
(4)
where j(a: fo) denotes the empirical class probability of fe applied It is easy to show that Sy is non-negative (see Appendix B for proof).
Additionally, we implemented 6 (5) and Kiy (6) based on the pretrained weights of the model before it was fine-tuned with our procedure, which we refer to as lg. Thereafter, the unsupervised symmetric pairwise selection function 6 implemented using pseudo-labels computed by the given pretrained model fe..
fe, + foo (xz))
otherwise,
and the confidence function K, is implemented using the standard softmax response, i.e..
(x) - max (f)(
(6)
where ((x]fe,) is the empirical class probabilities of fe, applied to and C' is the number of classes. Finally, the optimization objective (3) is implemented using the standard cross-entropy loss, CE(X, Yilfo) _ 1, El log Pv. (2; fo), as well the proposed Sy, oy and K.y.
C(X,, Xulfo Sy
CE(X,, Yi fol A Boaas S1. k,)
(7)
where À E R is regularization hyperparameter.
4.2 A Large Margin Analogy of TransBoost
SVM
T-SVM
Red Train Instances
Blue Train Instances
Test Instances
CrrensBoost 0.1796
Trannsooost 0.05167
Figure toy quantitative example indicates our Drams encourages hehavior similar to TSVM.
TransBoost is inspired by TSVM, in which large margin principle is applied to the given (unlabeled) test sample to leverage the knowledge of this specific test set. In Figure we present toy quantitative (graphical) example indicating that transductive loss component, TranaBooss: encourages behavior similar to TSVM. Consider Figure [ (left) showing the decision boundary obtained by SVM for the given training set (and ignoring the test points). Using SVM to implement 5 (to determine if two test points are the same class), we get that TrannsBoos = 0.1796 (shown at the bottom). Clearly, this loss value is significantly larger than the corresponding loss (0.05167) obtained when using TSVM (right), where large margin principle of test instances is applied.
5 Empirical Study
In this section we present a comprehensive empirical study of TransBoost. We begin by describing our experimental design and then present our studies in detail.
<5)( 0 iff fo):)fo) and fox one-hot vectors where the indicate different classes.
5

---

5.1 Experimental Design and Details
This section describes the experimental details: datasets used, architectures used, and our TransBoost's procedure details and hyperparameters.
Datasets and Preprocessing. Most of our study of TransBoost is done using the well-known ImageNet- k ILSSVRC-202 dataset [19]. which contains 1 1,281, 166 training instances and 50,000 test instances in 1,000 categories. Preprocessing for training is standard and includes resizing and random cropping to 224 x 224, followed by random horizontal flip. For testing, the images are resized and center cropped. In some of our experiments involving advanced architectures from the Timm repository [20], we applied the test augmentations (at test time only) using its code from the Timm repository. Additionally, we investigated the effectiveness of TransBoost on the Food-10 -01 21]. CIFAR-10 and CIFAR- 100 [22], SUN-397 [23], Stanford Cars [24]. FGVC Aircraft [25], the Describable Textures Dataset (DTD) 22 and Oxford 102 Flowers [27]. For each of these datasets, we applied the same preprocessing as in ImageNet, with the exception of CIFAR-10 10 and CIFAR-100) where we used random cropping with padding (cropping to x2), followed by a random horizontal flip, a single RandAugment (28 operation and MixUp (29].
Architectures. We examined various inductive (standard) pretrained architectures as baselines for TransBoost. Specifically, we used a variety of ResNet [30] architectures: ResNet18, ResNet34, ResNet50, ResNet101 and 55s which were pretrained using the standard PyTorch training recipe 31. We also considered ResNet50-StrikesBacd [32], which is the ResNet50 architecture trained with Timm's 's [20] advanced training recipe. Additionally, we show results with more advanced architectures spanning various families: EfficientNetBO [33 and MobileNetV3--L [8], ConvNext-T 77 and the Transformer models ViT-S 99 and Swin-T [4], which were pretrained with advanced procedures of Timm's 's library 20 in accordance to their original papers.
Details the TransBoost Fine-Tuning Procedure. Throughout all TransBoost's experiments, we followed Algorithm| [ and our functions were implemented as described in Section 4.] We used the SGD optimizer (Nesterov) with momentum of 0.9, weight decay of 10-A, and batch size of 1024 (consisting of 512 labeled training instances and 512 unlabeled test instances). Unless otherwise specified, the learning rate was fixed 10-3 with no warmup for 120 epochs. The regularization hyperparameter of our loss in all our experiments was fixed to X = 2; see Equation 7 Our hyperparameters were fine-tuned based on a validation set that was sampled from the training set over ImageNet.
5.2 ImageNet Experiments: Transductive vs. Inductive
This study examines the transductive performance of TransBoost applied to various pretrained models (all described in Section and the transductive performance of the other baselines (SSL and t-FSL. versus the inductive performance of all baselines. Table I presents all the results. We note that some of these results have been already highlighted in Figure
Table M is divided into several horizontal sections. In the first section we present TransBoost applied to a number of ResNet architectures. In the second section, we discuss modern architectures, including two types of Vision Transformers, and also ConvNext-T as representative of the ConvNext family, which presently achieves SimCLR and SimCLRv2 SSL methods (see Appendix for more details on how we trained them in transduction). We note that SimCLRv2 presently dominates SSL performance over ImageNet according to 14. The last table section considers two popular t-FSL methods (which are described in Section 3) The column structure of Table D is straightforward but the most important column that deserves some explanation is the Inductive / Transductive column t4th column) in which we compare the transductive performance of each experiment to the relevant inductive fully supervised performance. For example, TransBoost is capable of increasing the ImageNet top-1 accuracy of VIT-S by 2.28%.
art results over ImageNet [14]. Next we present the
Table and Figure demonstrate that fine-tuning with TransBoost consistently and significantly improves the inductive top-1 accuracy performance in all cases. Even the tiny version of the top performing ConvNext architecture family (presently the state-of-the-arr according to 14] is improved by TransBoost. Surprisingly, ViT-S performed better than ConvNext-T when taking advantage of the given test samples using TransBoost, indicating that members of the VIT family could be the best transductive performers on ImageNet. A second interesting fact is that ViT-S and Swin-T

---

Params (M)
Inductive / Transductive
Method
Architecture
Topl Ace. (%)
Improv. (%)
ResNet (30
architectures that were pretrained using standard simple procedures:
ResNet18
11.69
69.76 73.36
+3.60
ResNet34
21.80
73.29 76.70
+3.41
TransBoost
ResNet50
25.56
76.15 / 79.03
+2.88
ResNet101
44.55
77.37 / 79.86
+2.49
ResNet152
60.19
78.33 / 80.64
+2.31
Advanced vision architectures that were pretrained using advanced procedures:
EfficientNetBO [3
05.29
77.70 78.60
+0.90
MobileNetV3-L.
05.48
75.78 76.81
+1.03
SosNet50-StrikesBace [32
25.56
80.38 / 81.15
+0.77
TransBoost
ConvNext-T 7
28.59
82.05 82.46
+0.41
ViT-S [9]
22.05
81.39 83,67
+2.28
Swin-T [34
28.29
81.38 82.16
+0.78
Baseline representatives of supi-vsper learning algorithms in transduction:
ResNet50
25.56
76.15 75.95
-0.20
SimCLR
ResNet152
60.19
78.33 / 78.00
-0.33
ResNet50
25.56
76.15 74.68
-1.47
SimCLRv2
ResNet152
60.19
78.33 76.64
-1.69
Baseline representatives of transductive few-shot learning algorithms:
Ent-min min 15
ResNet50
25.56
76.15/ 77.55
+1.40
TIM-GD 16
ResNet50
25.56
76.15 71.51
-4.64
Table Inductive vs. Transductive. A comprehensive analysis of the transductive performance on ImageNet compared to the standard inductive (fully supervised) performance. TransBoost's performance is highlighted in green, SSL's performance highlighted in blue, and t-FSL's performance is highlighted in yellow.
perform similarly in the inductive setting while the ViT-S top-1 accuracy gain is improved by almost x3 relative to Swin-T. Could this advantage related to the architecture? Interestingly, TransBoost also improves the performance of DesNee55--StrikesBaaae which is the standard ResNer50 architecture trained using a sophisticated training procedure [32] that boosts its top-1 accuracy by +4.2%. Surprisingly, TransBoost adds almost +1% on top of this phenomenal improvement. Another striking related result that TransBoost improves the standard ResNet50 so that it achieves performance close to Mareeet55-trrieBaa.a. despite the fact that procedure is much simpler than the sophisticated training procedure of [32]
Consider the first section of Table | (the ResNets section). While TransBoost always improves the inductive baselines, is evident that its relative advantage monotonically decreases with the model size/performance. For example, the best transductive improvement is achieved for ResNet18 yielding top-1 accuracy gain +3.6% and resulting top-1 accuracy of 73.36% With respect to the other related baselines, TransBoost consistently outperforms both the SSL and [-FLL baseline algorithms. Comparing the results of SimCLR to the results of SimCLRv2, we see that SimCLR outperforms SimCLRv2 in the transductive setting (they mainly differ by the additional stage of knowledge distillation 36 that added to SimCLRv2), which suggests that the stage of knowledge distillation can hinder the transductive performance. We hypothesize that distilling knowledge from test instances leads to overfitting of incorrect class predictions that lead to model confusion. Finally, when evaluating the t-FSL representatives, observe that Ent-min improves its inductive baseline while TIM-GD (which an upgraded version of Ent-min) is inferior to its inductive baseline.
5.3 TronsBoost's Performance on Additional Vision Datasets
We now examine the performance of TransBoost across eight other datasets (all described in Sec- 3.II. Throughout this study, we start with ResNet50 architecture pretrained on ImageNet (using PyTorch standard training recipe). Then we apply inductive fine-tuning for each specific dataset and, finally, we apply our TransBoost procedure (see Appendix D.1 for more details).
This ResNet18 transductive performance even outperforms the current state- of thethh-rrt inductive performance 14 for ResNet18 ImageNet (73.19%) [35), which achieved with fancy training recipe.
7

---

97.61
97.85
85.29
96
97.6
+2.99
95.94
84
+2.52
83.8
84.3
94.86
+2.69
82.77
+5.34
93.25
92
90.8
79.88
90
+3.65
76
+1.7
87.15
74,79
86
74
CIFAR-10
Cars
Flowers
SUN
CIFAR-100
Aircraft
DTD
Food
Figure TransBoost's transductive performance using ResNet50 (green) on various vision datasets.
The results are visually depicted in the graph of Figure and are also shown in Table S.I in the Appendix. Clearly, TransBoost outperforms its inductive baseline all cases. The improvement is large and significant in all datasets with the exception of CIFAR-10 where the improvement is minor. The most prominent result is obtained for FGVC Aireraft dataset, where we observe a striking +5.34% top-1 accuracy gain from 78. 78.46% to 83.8%. For more in-depth discussion, we refer the reader to Appendix E.
5.4
Transductive Performance Improves with Test Set Size
Improv. Top-1 Acc. (%)
+4.0.3.9%..67
+3.42
+3.18-3.2543.16
+3.38
10% Test
42.89
+2.88
25% Test
-2.43
+2.34
+2.34
+1.8
50% Test
100% Test
5%
10%
20%
100%
ImageNet Train Fraction
Figure 4: TransBoost's transductive performance using ResNet50 on ImageNet training and test subsets.
In this study, we consider various transductive settings, where each setting is characterized by certain training size and a certain test set size. We are interested in examining the final ImageNet transductive behavior of ResNet50 as function of these sizes. The sizes our training sets are: 5%, 10%, 20%, 100%; and for testing sets are: 10%, 25%, 50%, 100%, all taken fraction of the original train/validation set sizes, respectively.
Consider Figure 4 showing the top-1 accuracy gains for all combinations (4 x 4 experiments in total). Importantly, we observe that whenever TransBoost uses the entire training set, its performance increases as the test set size grows. This strongly indicates that TransBoost leverages the test set as a whole, desirable property of a transductive learning algorithm. On the other hand, and quite surprisingly, when TransBoost only used 5% of the training set, increasing the test set size worsened its performance. small training set) that leads the models to wrongly analyze the relationships (similarities in our case) between pairs. This overall bad effect increases as the test set size expands. For a detailed description/ddieussioon of this section's experiments, Appendix E.
hypothesize that this type behavior a result of poor prior knowledge (very
5.5 Bringing the Like Together or Separating the Difference?
TransBoost driven by Equation (2) whose informal desideratum is: revise the model as to take apart the representations of two instances that are likely (K) to belong to different classes (6), which currently appear to be similar (S). Rather than, or addition to separating instances that are likely to be different, we could consider grouping together instances that are likely to belong to the same class. this section we summarize our experiments apply TransBoost for this alternative objective and for combination of both objectives. In study used the ResNet50 architecture,

---

which was pretrained on ImageNet (using Py Tyrch''s 's standard training recipe). The transductive performance of TransBoost was analyzed using three variants of the transductive loss component: (1) the proposed separation component, TransBoou X S 8,, Kif), as in Equation 7 (2) "reciprocal" instantiation of this objective namely, IrransBoos (X,l 2 Sy, 1 8, k which brings together similar instances; and (3) a variant that considers both these objectives together. The results are
presented in Table
Inductive ''ransducttve
Transductive Loss
Topl Acc. (%)
Improv. (%)
TransBooss [X,, (7
76.15 / 79.03
+2.88
TransDoost ( Xulv2 S k5) CrramsBoos ( (y 87, R5) + Lrmmmoou (X.. 11,1-8fm1
76.15 74.64
-1.51
76.15 79.00
+2.85
Table 2: Performance transductive loss variants using ResNet50.. Our proposed loss is highlighted in green.
Clearly (but not significantly) the best performance is achieved using proposed implementation (row highlighted in green). We chose our proposed loss component over the combination of both objectives because it is simpler. Interestingly, the reciprocal variation significantly degrades the top-1 performance.
Concluding Remarks
We presented TransBoost, novel and powerful transductive fine-tuning procedure that can be effi- ciently applied on any pretrained model. TransBoost a deep transductive classification algorithm that is inspired by a large margin principle such as the TSVM algorithm. Strikingly, TransBoost con- sistently and significantly improves the inductive ImageNet top-1 performance of many architectures including the most advanced ones, leading to state-of- the-art transductive performance. Moreover, TransBoost is effective on a broad range of image classification datasets.
As described in Algorithm 1 TransBoost is general optimization procedure that can be implemented in a range of ways depending on the similarity function (S), the selection function (6), and the confidence function (ks) We instantiated these functions simply and straightforwardly, A question for future research may be to ask whether more sophisticated choices will lead to better transduction performance. The ViT-S architecture achieved the best transductive classification performance in our experiments and accrued the largest gain relative to induction among advanced architectures. would be interesting explore the question of how this Transformer architecture facilitates this gain. In general, we can ask what is the best architecture for transductive classification. Finally, it would be very interesting
whether such large performance gains are also sustained in NLP applications.
Finally, our results clearly demonstrate that transductive learning can lead huge accuracy gains. Thus, it would be wasteful to use standard induction in settings where transduction is applicable.
Acknowledgments
This research was partially supported by the Israel Science Foundation, grant No. 710/18.
References
[1] Philip Derbeko, Ran El- Yaniv, and Ron Meir. Explicit learning curves for transduction and application to clustering and compression algorithms. Journal of Artificial Intelligence Research, 12:117-142, 2004.
[2] Ran El-Yaniv and Dmitry Pechyony. Transductive rademacher complexity and its applications. Journal of Artificial Intelligence Research, 35:193-234, 2009.
[3] Ran El-Yaniv. Dmitry Pechyony, and Vladimir Vapnik. Large margin vs. large volume in transductive learning. Machine Learning, 7203):17-188, 2008.
[4] Vladimir Vapnik, The nature of statistical learning theory. Springer science business media,
1999.

---

[5] Thorsten Joachims et al. Transductive inference for text classification using support vector machines. In lcml, volume 99, pages 200-209, 1999.
[6] Corinna Cortes and Vladimir Vapnik. Support vector machine. Machine learning, 20(3):273- 297, 1995.
[7] Zhuang Liu, Hanzi Mao, Chao-Yual Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. arXiv preprint arXiv:2201.035555 2022.
[8] Andrew Howard, Mark Sandler, Grace Chu, Liang-Chieh Chen, Bo Chen, Mingxing Tan, Weijun Wang, Yukun Zhu, Ruoming Pang, Vijay Vasudevan, et al. Searching for mobilenetv3. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 1314-1324, 2019.
[9] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021.
[10] Vladimir Vapnik. Estimation of dependences based empirical data. Springer Science & Business Media, 2006.
[11] Ting Chen, Simon Korbblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference machine learning, pages 1597-1607. PMLR, 2020.
[12] Ting Chen, Simon Korbllith, Kevin Swersky, Mohammad Norouzi, and Geoffrey Hinton. Big self-supervised models strong semi-supervised learners. Advances in neural information 2020.
processing systems, 33:22243-
[13] Mahmoud Assran, Mathilde Caron, Ishan Misra, Piotr Bojanowski, Armand Joulin, Nicolas Ballas, and Michael Rabbat. Semi-supervised learning of visual features by non parametrically predicting view assignments with support samples. In Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision, pages 8443-8452, 2021.
[14] Papers with code: The latest in machine learning. htttp:://papersitthenn
com
[15] Guneet Singh Dhillon, Pratik Chaudhari, Avinash Ravichandran, and Stefano Soatto. A baseline for few-shot image classification. In International Conference on Learning Representations, 2019.
[16] Malik Boudiaf, Imtiaz. Ziko, Jérôme Rony, José Dolz, Pablo Piantanida, and Ismail Ben Ayed. Transductive information maximization for few-shot learning. Advances Neural Information Processing Systems, 33:2445-245 2020.
[17] Michalis Lazarou, Tania Stathaki, and Yannis Avrithis. Iterative label cleaning for transduc- tive and semi-supervised few-shot learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8751-8760, 2021.
[18] Few-shot classification leaderboard.
org
[19] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, al. Imagenet large scale visual recognition challenge. International journal of computer vision, 1 115(3):211-252, 2015.
[20] Ross
Wightman. Pytorch image models.
ponorcchimaagemmddeee 2022.
[21] Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool.
discriminative components with random forests. In European conference on computer vision, pages 446-461. Springer, 2014.
[22] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[23] Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey to zoo. In 2010 IEEE computer society conference on computer vision and pattern recognition, pages 3485-3492. IEEE, 2010.
[24] Jonathan Krause, Jia Deng, Michael Stark, and Li Fei-Fei. Collecting large-scale dataset of fine-grained cars. 2013.
10

---

[25]
Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine- grained visual classification of aircraft. arXiv preprint arXiv:1306.575,, 2013.
[26]
Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3606-3613, 2014.
[27] Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over large number of classes. In 2008 Sixth Indian Conference on Computer Vision, Graphics & Image Processing, pages 729. IEEE, 2008
(28) Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoe V Le. Randaugment: Practical automated data augmentation with a reduced search space. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition Workshops, pages 702-703, 2020.
[29] Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. International Conference on Learning Representations, 2018.
[30] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[31] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019.
[32] Ross Wightman, Hugo Touvron, and Hervé Jégou. Resnet strikes back: improved training procedure in timm. arXiv preprint 10.00476, 2021.
[33] Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In International conference on machine learning, pages 6105-6114, 14. PMLR, 2019.
[34] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10012-10022, 2021.
[35] Zhiqiang Shen and Marios Savvides. Meal v2: Boosting vanilla resnet-50 to 80%+ top-1 accuracy on imagenet without tricks. arXiv preprint arXiv:2009.08445 2020.
[36] Geoffrey Hinton, Oriol Vinyals, and Jeffrey Dean. Distilling the knowledge in neural network.
In NIPS Deep Learning and Representation Learning Workshop, 2015.


Reviews:
[{"review_id": "lbrhEu-PIwY", "reviewer": "Reviewer_Sp8K", "summary": "The paper proposes TransBoost to deal with deep transductive learning. The method follows the large margin principle and includes the similarity and confidence of two instances in the loss function. The effectiveness of this method is examined by using a variety of pre-trained neural networks, datasets, and baseline models. It shows good performance compared to inductive classification as well as versus SSL and t-FSL methods", "questions": "1. Some paragraphs are hard to read. For example, in the 3rd paragraph in Section 4, what does it mean: we incentivize test sample pairs that are likely to be different in their classes to also be different in their empirical class probabilities while preserving the prior knowledge of $f_{\\theta}$?\n2. The toy example in Section 4.2 is unclear:\n(a) How is the p(x|f) calculated with SVM?\n(b) Although TransBoost loss is lower in T-SVM, does the decision boundary of T-SVM provides better separation of the test instances? So what is the distribution of test instances?\n3 Typo?\n(a) Section 5.5: separating the different -> separating the difference\n(b) Figure 2: ... example indicating ... -> ... example indicates ...", "limitations": "N/A", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\nIt provides extensive experiments on multiple neural network architectures and datasets and shows good improvement.\n\nWeakness:\nThe paper is not in the submission format, there is no line number. \nClarity: some paragraphs are hard to read.\nSignificance: the author doesn't provide a strong motivation for why their work is important", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "KyIvPRrZs1", "reviewer": "Reviewer_GxTe", "summary": "Paper proposes TransBoost, a transductive learning loss function that improves performance in transductive image classification in domains where large datasets and test sets are available. The work draws inspiration from TSVM and proposed a transductive loss regularization term that discriminates unlabelled examples that have similar class probabilities but shouldn't. Extensive experiments demonstrate that TransBoost improves performance on various architectures, datasets, and settings. Ablation studies provide interesting insights into the behaviour of TransBoost under different conditions.", "questions": "Suggestions are included in the above sections. An overall strong submission that could be further strengthened by additional analysis of TransBoosts uneven behaviour with respect to architectures.", "limitations": "Yes, the authors provide an in-depth analysis of the technical limitations of the work and provide directions for future research. There is, however, no explicit discussion of societal impacts. I believe the work could benefit from a concise analysis of its societal impact and if any could potentially be negative.", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- Paper is incredibly well-written and very easy to follow.\n- TransBoost is an interesting and relatively novel idea. It effectively bridges the gap in performance from t-FSL methods that have been widely developed. The algorithmic choices are sound and well-motivated.\n- Experiments are extensive and provide sufficient evidence to establish state-of-the-art performance.\n\n(Small )Weaknesses:\n- Certain results suggest that TransBoost is more applicable to some architectures than others. Although performance improvements are universal, some methods improve much more, and some even surpass their otherwise more inductively powerful counterparts. This suggests that TransBoost is algorithmically better suited for certain architectures, but the reasoning is primarily speculated. It would be useful to see further analysis as to why for instance, Transformer based architectures benefit from TransBoost?\n- One would presume that alternatively modifying the loss function to group together similar samples would benefit performance, albeit not to the same extent. However, the observation that it degrades performance may suggest that transductive learning can produce fragile optimization manifolds that should be carefully designed. Further elaboration on this front could be useful.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Fq1ZdERlH-s", "reviewer": "Reviewer_XZ2m", "summary": "This paper investigates deep transductive learning where both a labeled training sample and an unlabeled test sample are provided at training time. The goal of deep transductive learning is to achieve high accuracy on the test set. To this end, the authors propose TransBoost as a procedure to fine-tune any deep neural model using a labeled training sample and an unlabeled test sample. Extensive experiments are conducted to verify the effectiveness of the proposed method.\n", "questions": "Please see details in Cons.  I would like to suggest the authors define a meaningful task. Eqn.(2) borrows ideas from TSVM and it would have a positive effect in test time training [1]\n\n[1] Sun et al. Test-Time Training with Self-Supervision for Generalization under Distribution Shifts. ICML 2020.\n", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Pros:**\n1. The paper is well-written and well-organized.\n2. Investigating conventional statistical machine learning in the context of deep learning is interesting and meaningful.\n3. Extensive experiments are conducted to verify the effectiveness of the proposed task and method.\n\n**Cons:**\n1. The task of deep transductive learning has a severe issue of overfitting target test set. Although Transboost does not use the true label of the test set, it implicitly uses pseudo labels, i.e. similarity of image pairs. In this sense, the performance would boost a lot in any deep models as shown in Fig.1.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept