SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We assume there exist  $B$  classes in the base session and  $C$  classes in an incremental session, e.g.,  $B = 60, C = 5$  in the CIFAR100 dataset. Without loss of generality, we only consider the base session and the first incremental session for simplification. Other incremental sessions can be obtained in the same way. Following the notations in section 3.1, the empirical prototype of  $i$ -th class can
be notated as  $c_{i}$ . Therefore, the base prototypes are  $c_{b}(0 \leq b \leq B - 1)$  and the new prototypes are  $c_{n}(B \leq n \leq B + C - 1)$ . As the base session contains sufficient classes and sufficient samples for each class, the model trained on the base session can capture the distribution of base classes and obtain well-calibrated prototypes for base classes. Generally, the empirical prototype of base classes can be regarded as approximately consistent with the expected class representation. However, due to the limited data in incremental sessions, the empirical prototypes of the novel classes are considered to be severely biased.
Based on the analysis of prototypes, we only calibrate biased prototypes in incremental sessions. Given a new class prototype  $c_{n}(B \leq n \leq B + C - 1)$ , the calibrated new prototype  $\bar{c}_{n}$  can be notated by:
$$
\bar {c} _ {n} = (1 - \alpha) c _ {n} + \alpha \Delta c _ {n} \tag {4}
$$
The calibration item  $\Delta c_{n}$  is a component of base prototypes. The hyperparameter  $\alpha$  controls the calibration strength of biased prototypes. Smaller  $\alpha$  means the calibrated prototype reflects more of the original biased prototype, while larger  $\alpha$  means the calibrated prototype heavily incorporates the base prototypes. Motivated by the observations in section 4.1, the similarity between well-calibrated base prototypes and ill-calibrated new prototypes contains auxiliary side information about the new classes. Therefore, we use weighted base prototypes to represent the calibration item  $\Delta c_{n}$  and enhance the discriminability of biased prototypes. Specifically, we compute the cosine similarity  $S_{b,n}$  between a new class prototype  $c_{n}$  and a base prototype  $c_{b}$ :
$$
S _ {b, n} = \frac {c _ {b} \cdot c _ {n}}{\| c _ {b} \| \cdot \| c _ {n} \|} \cdot \tau \tag {5}
$$
where  $\tau (\tau >0)$  is the scaling hyperparameter controlling the weight distribution's sharpness. The weight of a new prototype  $c_{n}$  with respect to a base prototype  $c_{b}$  is the softmax results over all base prototypes:
$$
w _ {b, n} = \frac {e ^ {S _ {b , n}}}{\sum_ {i = 0} ^ {B - 1} e ^ {S _ {i , n}}} \tag {6}
$$
Finally, the calibration of biased prototypes of new classes can be formulated as follows:
$$
\bar {c} _ {n} = (1 - \alpha) c _ {n} + \alpha \Delta c _ {n} = (1 - \alpha) c _ {n} + \alpha \sum_ {b = 1} ^ {B - 1} w _ {b, n} c _ {b} \tag {7}
$$
Notably, the above prototype rectification procedure is a training-free calibration strategy because it does not introduce any learning component or training parameters. Figure 3a gives an intuitive description of the calibration effect.
5.2 Effect of calibrated prototypes
Intuitively,  $w_{b,n}$  is larger when a new prototype  $c_{n}$  and base prototype  $c_{b}$  are more similar. Given a new prototype  $c_{n}$ , we assume the most similar base prototype with  $c_{n}$  is the base prototype  $c_{n}^{base}$ . Therefore, given a new prototype  $c_{n}, \sum_{b=1}^{B-1} w_{b,n} c_{b} \approx c_{n}^{base}$  when the scaling hyperparameter  $\tau$  is large enough. From the perspective of a biased prototype, a calibrated prototype  $\bar{c}_{n}$  will be aligned to its most similar base prototypes  $c_{n}$  with a proper  $\tau$ .
To further comprehend the effect of TEEN on the predictions of base and new classes respectively, we define three types of test samples according to whether the prediction results change after TEEN: with unchanged predictions (i.e., UC samples), with prediction going from right to wrong (i.e.,  $\mathbf{R} \rightarrow \mathbf{W}$  samples), with predictions going from wrong to right (i.e.,  $\mathbf{W} \rightarrow \mathbf{R}$  samples). We analyze the specifics of these three types of samples in detail.
Intuitively, some calibrated prototypes of new classes are aligned to base prototypes and reduce the discriminability of base prototypes. Oppositely, aligning the biased prototype to most similar base prototypes can calibrate the prediction of new classes. As shown in Table 3b, we observe the  $\mathbf{W} \rightarrow \mathbf{R}$  samples are mainly from new classes and the  $\mathbf{R} \rightarrow \mathbf{W}$  samples are mainly from base classes. Besides, extensive comparison results on the benchmark datasets (i.e., Table 3 and Figure 4) show that the negative effect of TEEN is negligible due to the significance of TEEN.
Table 3: Detailed average accuracy of each incremental session on miniImageNet dataset. Please refer to the supplementary for results on CUB200 and CIFAR100. The results of compared methods are cited from [42, 57, 58].  $\uparrow$  means higher accuracy is better.  $\downarrow$  means lower PD is better.
Method	Accuracy in each session (%) ↑	PD ↓	Δ PD
0	1	2	3	4	5	6	7	8
iCaRL [35]	61.31	46.32	42.94	37.63	30.49	24.00	20.89	18.80	17.21	44.10	+22.65
EEIL [4]	61.31	46.58	44.00	37.29	33.14	27.12	24.10	21.57	19.58	41.73	+20.28
Rebalancing [18]	61.31	47.80	39.31	31.91	25.68	21.35	18.67	17.24	14.17	47.14	+25.69
TOPIC [42]	61.31	50.09	45.17	41.16	37.48	35.52	32.19	29.46	24.42	36.89	+15.44
Decoupled-NegCosine [26]	71.68	66.64	62.57	58.82	55.91	52.88	49.41	47.50	45.81	25.87	+4.42
Decoupled-Cosine [43]	70.37	65.45	61.41	58.00	54.81	51.89	49.10	47.27	45.63	24.74	+3.29
Decoupled-DeepEMD [56]	69.77	64.59	60.21	56.63	53.16	50.13	47.79	45.42	43.41	26.36	+4.91
CEC [57]	72.00	66.83	62.97	59.43	56.70	53.73	51.19	49.24	47.63	24.37	+2.92
FACT [58]	72.56	69.63	66.38	62.77	60.6	57.33	54.34	52.16	50.49	22.07	+0.62
TEEN	73.53	70.55	66.37	63.23	60.53	57.95	55.24	53.44	52.08	21.45
(a) miniImageNet
(b) CUB200
Figure 4: Top-1 average accuracy on all seen classes in each incremental session. We annotate the performance gap between TEEN and the runner-up method by  $\uparrow$ .
In this section, we first introduce the main experiment details of FSCIL in section 6.1, which include the implementation and performance detail. Subsequently, we introduce the FSL performance of TEEN in section 6.2, which consistently shows the effectiveness of TEEN. Finally, we evaluate TEEN by a comprehensive ablation study in section 6.3.
6.1 Main experimental details of FSCIL
6.1.1 Implementation Details
Datasets and baseline details: Following previous methods [42, 57, 58], we evaluate TEEN on CIFAR100 [23], CUB200-2011 [44], miniImageNet [38]. We keep the dataset split consistent with existing methods [57, 58]. Notably, each benchmark dataset is divided into subsets containing nonoverlapping label space. For example, CIFAR100 is divided into 60 classes for the base session and the left 40 classes are divided into eight 5-way 5-shot few-shot classification tasks. To validate the performance of TEEN, we compare TEEN with popular CIL methods [35, 4, 18], FSL methods [26, 56, 43], and FSCIL methods [42, 57, 58]. Please refer to the supplementary material for more datasets and baseline details.
Training details: All experiments are conducted with PyTorch [32] on a single NVIDIA 3090. The training of the feature extractor uses vanilla cross-entropy loss as the objective function. It does not evolve any extra complex pretraining module [58, 59, 10, 57], thus making TEEN more efficient and elegant. In addition, we adopt the cosine similarity to measure the similarity between the instances and class prototypes. Following [42, 57, 58], we use ResNet20 [16] for CIFAR100, pre-trained ResNet18 [16] for CUB200 and randomly initialized ResNet18 [16] for miniImageNet. All compared methods use the same backbone network and initialization for a fair comparison. We set  $\alpha = 0.5$ ,  $\tau = 16$  for miniImageNet and CUB200,  $\alpha = 0.1$ ,  $\tau = 16$  for CIFAR100. We train the feature extractor on CUB200 with a learning rate of 0.004, batch size of 128, and epochs of 400. Please refer to the supplementary for more training details on CIFAR100 and miniImageNet.
Table 4: Detailed results of HMean and NAcc on miniImageNet. The best results are in bold and the runner-up results are in underlines. The  $\Delta$  measures the performance gap between the best and second-best results on the corresponding session. Due to space limitations, the performance on only six incremental sessions is presented. Please refer to the supplementary for more detailed results on CUB200 and CIFAR100.
Session	1	2	3	4	5	6
HMean/NAcc	HMean	NAcc	HMean	NAcc	HMean	NAcc	HMean	NAcc	HMean	NAcc	HMean	NAcc
CEC [57]	30.72	19.60	30.05	19.10	29.86	19.00	29.41	18.65	27.15	16.88	27.36	17.07
FACT [58]	30.60	19.20	27.84	17.10	25.89	15.67	23.85	14.20	22.01	12.92	20.65	12.00
TEEN	50.04	38.00	46.67	34.60	44.72	32.67	43.53	31.55	41.75	29.80	39.22	27.37
Δ	+19.32	+18.4	+16.62	+15.5	+14.86	+13.67	+14.12	+12.9	+14.6	+12.92	+11.86	+10.3
6.1.2 Comparison results
In this section, we conduct overall comparison experiments on different performance measures. These different performance measures focus on different aspects of the methods. For example, the widely-used average accuracy across all classes (i.e., AvgAcc) is difficult to reflect the performance of new classes because the base classes take a large percentage of all classes (e.g., 60 base classes of 100 classes in CIFAR100). Following [58], the Harmonic mean (i.e., HMean) is used to evaluate the balanced performance between the base and new classes. Besides the above performance measures, we also additionally evaluate the different methods of their performance on new classes (i.e., NAcc). Following [42, 57, 58], we also measure the degree of forgetting by the Performance Dropping Rate (i.e., PD). The PD is defined as  $PD = Acc_0 - Acc_{-1}$ , i.e., the average accuracy dropping between the first session (i.e.,  $Acc_0$ ) and the last session (i.e.,  $Acc_{-1}$ ). The detailed comparison results of PD and AvgAcc are reported in Table 3, and the detailed comparison results of NAcc and HMean are reported in Table 4. These experimental results from different performance measures all demonstrate the effectiveness of TEEN. Besides, significantly lower FPR and TBR in Table 1 and Table 2 also verify the effective calibration of TEEN.
Notably, many previous state-of-the-art FSCIL methods (e.g., [57, 58]) usually design complex pretraining algorithms to enhance the extendibility of feature space, which may harm the discriminability of base classes. This elaborate pretraining stage may lead to an inconsistent but negligible performance in the base session. Besides, PD $\downarrow$  and  $\Delta$ PD in Table 3 are not affected by the pretrained results and also show TEEN outperforms previous state-of-the-art methods.
6.2 Comparison results of FSL
FSL can be approximated as measuring only the performance of the new classes in the first incremental session of FSCIL. Besides, FSL itself also faces the challenge of biased class prototypes due to the few-shot data. Therefore, we validate TEEN in the setting of FSL and report the compared results in Table 5. To ensure a fair comparison, we strictly followed the experimental setup of [51]. The comparison results in Table 5 show
Table 5: Few-Shot Leaning performance of classification accuracy  $(\%)$  on miniImageNet and CUB. The results of compared methods are cited from [51]. 5w1s and 5w5s mean 5way-1shot and 5way-5shot, respectively. The best results are in bold and the runner-up results are in underlines.
Methods	miniImageNet	CUB
5w1s	5w5s	5w1s	5w5s
ProtoNet [40]	54.16	73.68	72.99	86.64
NegCosine [26]	62.33	80.94	72.66	89.40
LR with DC [51]	68.57	82.88	79.56	90.67
TEEN	65.70	83.11	81.44	91.04
that TEEN can easily outperform the previous state-of-the-art method [51] in several experimental settings. Notably, TEEN does not involve any additional training cost when recognizing new classes, and the only time cost lies in feature extraction. Therefore, once the sample features are extracted, the inference cost of TEEN can be considered negligible compared to previous methods that require heavy training.
The influence of  $\alpha$  and  $\tau$ : Notably, the proposed TEEN does not involve additional training-based modules or procedures after pretraining on base classes. When TEEN incrementally learns new classes, only the scaling temperature  $\tau$  and the coefficient of calibration item  $\alpha$  need to be determined.
(a)  $\alpha$  and  $\tau$  w.r.t AvgAcc
(b)  $\alpha$  and  $\tau$  w.r.t NAcc
Figure 5: (a) and (b) show the influence of  $\alpha$  and  $\tau$  on the last session's average accuracy in all classes (i.e., AvgAcc) and new classes (i.e., NAcc), respectively. (c) and (d) show the influence of similarity-based weight in TEEN on the last session's average accuracy in all classes (i.e., AvgAcc) and new classes (i.e., NAcc), respectively. It demonstrates that replacing the softmax-based weight with a hard one-shot weight will drastically reduce the performance of TEEN.
(c) SimTEEN w.r.t AvgAcc
(d) SimTEEN w.r.t NAcc
Specifically, we select  $\tau$  from  $\{8,16,32,64\}$  and  $\alpha$  from  $\{0.1,0.2,0.3,0.4,0.5,0.6,0.7,0.8,0.9\}$ . Figure 5a and Figure 5b show how the hyper-parameters  $\alpha$  and  $\tau$  influence the average accuracy on all classes and new classes, respectively. It also demonstrates that a proper  $\alpha$  can improve the performance of new classes, which confirms the base prototypes can benefit the performance of new classes. Compared with larger  $\tau$ , relatively smaller  $\tau$  can smooth the weight distribution. The smaller  $\tau$  with stronger performance further confirm the effectiveness of utilizing the abundant semantic information from base classes.
The influence of similarity-based weight: To further validate the effectiveness of the semantic similarity between the base and new classes, we remove the Equation 6 and directly align the new prototypes to their corresponding  $K$  most similar base class prototypes, i.e.,  $\bar{\mathbf{c}}_n = \alpha \mathbf{c}_n + (1 - \alpha)\sum_{k=1}^{K}\mathbf{c}_k$ . We denote this Simple version of TEEN without similarity-based weight as SimTEEN. We select  $K$  from  $\{1,3,5,7,9\}$ . Figure 5c and Figure 5d show how the similarity-based weight (i.e., Equation 6) influences the average accuracy on all classes and new classes, respectively. Notably, as shown in Figure 5d, there is a significant drop, particularly on new classes, after removing the similarity-based weight. This phenomenon confirms that TEEN can achieve calibration of new prototypes with the help of well-represented semantic similarity between the base and new classes.
Few-shot class-incremental learning is of great importance to real-world learning scenarios. In this study, we first observe existing methods usually exhibit poor performance in new classes and the samples of new classes are easily misclassified into most similar base classes. We further find that although the feature extractor trained on base classes can not well represent new classes directly, it can properly represent the semantic similarity between the base and new classes. Based on these analyses, we propose a simple yet effective training-free prototype calibration strategy (i.e., TEEN) for biased prototypes of new classes. TEEN obtains competitive results in FSCIL and FSL scenarios.
Limitations: The FSCIL methods mentioned in this paper all select the base and new classes from the same dataset. In other words, current FSCIL methods assume the model pre-trains in the same domain, increasing the restrictions on pre-training data collection. Therefore, a more realistic scenario is to pre-train on a dataset that is independent of the subsequent data distribution and then perform few-shot class-incremental learning on a target dataset that we expect. The intricate problem of cross-domain few-shot class-incremental learning will be thoroughly investigated in our future works.
This work is partially supported by National Key R&D Program of China (2022ZD0114805), NSFC (62376118, 62006112, 62250069), Young Elite Scientists Sponsorship Program of Jiangsu Association for Science and Technology 2021-020, Collaborative Innovation Center of Novel Software Technology and Industrialization.
A.1 Implicit Semantic Similarity
Notably, TEEN demonstrates that the off-the-shelf feature extractor is already capable of representing semantic similarity without the need for any additional training costs or auxiliary side information (e.g., the textual information of the class name [8]).
A.2 Efficiency
The only time cost in TEEN is the pre-training cost on the base classes. Furthermore, the training cost of the vanilla cross-entropy loss function is relatively small compared to other more complex training paradigms. Additionally, the calibration in TEEN does not evolve any training cost and update procedure. Therefore, the cost of incrementally learning new classes of TEEN can be considered negligible compared with existing methods.
A.3 Combination with better representation
In addition to combining with existing prototype-based methods, TEEN can also be combined with more powerful pre-trained models (e.g., self-supervised pre-trained model [7]). Notably, the recent popular vision-language models (e.g., CLIP [34]) can also be seen as a powerful pre-trained model and provide better representation.
B A Closer Look at FSCIL
In this section, we show the observations (i.e., Section 3 of the main paper) on more datasets. As shown in Table 10, the existing prototype-based FSCIL methods are extremely prone to misclassify new classes into base classes. Considering the misclassified instances of base classes, we quantitatively show that the samples of new classes are usually misclassified into base classes. These observations in the additional benchmark dataset are consistent with observations in the main paper, which further confirms the universality of our observation.
To further understand the aforementioned observation, we further analyze the question What base classes are the new classes incorrectly predicted into? and come to our second observation: The prototype-based classifier misclassifies the new classes to their corresponding most similar base classes with a high probability. The detailed analysis results in Table 11 further verify the correctness of our observation.
Table 6: Detailed average accuracy of each incremental session on CIFAR100 dataset. The results of compared methods are cited from [42, 57, 58].  $\uparrow$  means higher accuracy is better.  $\downarrow$  means lower PD is better.
Method	Accuracy in each session (%)↑	PD ↓	Δ PD
0	1	2	3	4	5	6	7	8
iCaRL [35]	64.10	53.28	41.69	34.13	27.93	25.06	20.41	15.48	13.73	50.37	+28.09
EEIL [4]	64.10	53.11	43.71	35.15	28.96	24.98	21.01	17.26	15.85	48.25	+25.97
Rebalancing [18]	64.10	53.05	43.96	36.97	31.61	26.73	21.23	16.78	13.54	50.56	+28.28
TOPIC [42]	64.10	55.88	47.07	45.16	40.11	36.38	33.96	31.55	29.37	34.73	+12.45
Decoupled-NegCosine [26]	74.36	68.23	62.84	59.24	55.32	52.88	50.86	48.98	46.66	27.70	+5.42
Decoupled-Cosine [43]	74.55	67.43	63.63	59.55	56.11	53.80	51.68	49.67	47.68	26.87	+4.59
Decoupled-DeepEMD [56]	69.75	65.06	61.20	57.21	53.88	51.40	48.80	46.84	44.41	25.34	+3.06
CEC [57]	73.07	68.88	65.26	61.19	58.09	55.57	53.22	51.34	49.14	23.93	+1.65
FACT [58]	74.60	72.09	67.56	63.52	61.38	58.36	56.28	54.24	52.10	22.50	+0.22
TEEN	74.92	72.65	68.74	65.01	62.01	59.29	57.90	54.76	52.64	22.28
Table 7: Detailed average accuracy of each incremental session on CUB200 dataset. The results of compared methods are cited from [42, 57, 58].  $\uparrow$  means higher accuracy is better.  $\downarrow$  means lower PD is better.
Method	Accuracy in each session (%) ↑	PD ↓	Δ PD
0	1	2	3	4	5	6	7	8	9	10
iCaRL [35]	68.68	52.65	48.61	44.16	36.62	29.52	27.83	26.26	24.01	23.89	21.16	47.52	+29.39
EEIL [4]	68.68	53.63	47.91	44.20	36.30	27.46	25.93	24.70	23.95	24.13	22.11	46.57	+28.44
Rebalancing [18]	68.68	57.12	44.21	28.78	26.71	25.66	24.62	21.52	20.12	20.06	19.87	48.81	+30.68
TOPIC [42]	68.68	62.49	54.81	49.99	45.25	41.40	38.35	35.36	32.22	28.31	26.26	42.40	+24.27
Decoupled-NegCosine† [26]	74.96	70.57	66.62	61.32	60.09	56.06	55.03	52.78	51.50	50.08	48.47	26.49	+7.36
Decoupled-Cosine [43]	75.52	70.95	66.46	61.20	60.86	56.88	55.40	53.49	51.94	50.93	49.31	26.21	+8.08
Decoupled-DeepEMD [56]	75.35	70.69	66.68	62.34	59.76	56.54	54.61	52.52	50.73	49.20	47.60	27.75	+9.62
CEC [57]	75.85	71.94	68.50	63.50	62.43	58.27	57.73	55.81	54.83	53.52	52.28	23.57	+5.44
FACT [58]	75.90	73.23	70.84	66.13	65.56	62.15	61.74	59.83	58.41	57.89	56.94	18.96	+0.83
TEEN	77.26	76.13	72.81	68.16	67.77	64.40	63.25	62.29	61.19	60.32	59.31	18.13
Table 8: Detailed results of HMean and NAcc on miniImageNet. The best results are in bold and the runner-up results are in underlines. The  $\Delta$  measures the performance gap between the best and second-best results on the corresponding session. Due to space limitations, the performance on only six incremental sessions is presented. Please refer to the supplementary for more detailed results on CUB200 and CIFAR100.
Session	1	2	3	4	5	6	7	8
HMean/NAcc	HMean	NAcc	HMean	NAcc	HMean	NAcc	HMean	NAcc	HMean	NAcc	HMean	NAcc	HMean	NAcc	HMean	NAcc
CEC [57]	47.04	34.80	40.35	28.10	36.01	24.13	36.49	24.65	36.56	24.84	37.07	25.40	36.41	24.83	35.49	24.08
FACT [58]	43.04	30.00	38.77	26.10	34.70	22.60	34.44	22.45	34.55	22.64	35.44	23.47	33.79	22.06	33.05	21.48
TEEN	47.77	34.82	42.99	30.10	40.19	27.53	39.52	27.00	39.66	27.28	39.65	27.37	37.71	25.57	37.11	25.13
Δ	+0.73	+0.02	+2.64	+2.00	+4.18	+3.40	+3.03	+2.35	+3.10	+2.44	+2.58	+1.97	+1.30	+0.74	+1.62	+1.05
C.1 Details of experiments
Dataset details: Following previous methods [42, 57, 58], we evaluate TEEN on CIFAR100 [23], CUB200-2011 [44], miniImageNet [38]. CIFAR100 [23] contains 100 classes and each class contains 500 images for training and 100 images for testing. The image size of CIFAR100 is  $3 \times 32 \times 32$ . The CUB200-2011 [44] is a widely-used fine-grained dataset and is also a benchmark dataset of few-shot image classification. It contains 200 classes and all images of these classes belong to birds. The miniImageNet [38] are sampled from the raw ImageNet [11] and contains 100 classes.
In FSCIL, each benchmark dataset is divided into different subsets. Each subset contains specific classes and the label space of different subsets is nonoverlapping. Specifically, CIFAR100 is divided into 60 classes for the base session and the remaining 40 classes are divided into eight 5-way 5-shot few-shot classification tasks. CUB200 is divided into 100 base classes for the base session and the remaining 100 classes are divided into ten 10-way 5-shot few-shot classification tasks. miniImageNet is divided into 60 base classes for the base session, and the remaining 40 classes are divided into eight 5-way 5-shot few-shot classification tasks. The splitting details (i.e., the class order and the selection of support data in incremental sessions) follow the previous methods [42, 57, 58]. Notably, no old samples are saved to assist in maintaining the discriminability of previous classes.
Baseline details: Following previous methods [42, 57, 58, 61], we compare TEEN with popular CIL methods, FSL methods, and FSCIL methods. For CIL methods, we select iCaRL [35], EEIL [4] and Rebalancing [18] as our baseline methods. For methods based on few-shot, we adopt Decoupled-NegativeCosine [26], Decoupled-Cosine [43] and Decoupled-DeepEMD [56] as our baseline methods. For FSCIL methods, we adopt TOPIC [42], CEC [57] and FACT [58] methods as our baseline methods.
Training details: The training of the feature extractor uses vanilla cross-entropy loss as the objective function. In addition, we adopt the cosine similarity to measure the feature of instances to class prototypes. Following [42, 57, 58], we use ResNet20 [16] for CIFAR100, pre-trained ResNet18 [16] for CUB200 and randomly initialized ResNet18 [16] for miniImageNet. All compared methods use the same backbone network and same initialization for a fair comparison.