SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Class-Agnostic Mask Generator Following Mask2Former [20], we use a pixel decoder enhanced with multi-scale deformable attention [103] to improve the features extracted from the frozen CNN-based CLIP backbone. The enhanced pixel features, together with a set of object queries [7, 83], are then passed through a series of mask decoders, where each consists of masked cross-attention [20], self-attention [81], and a feed-forward network. The resulting segmentation logits are obtained by performing a matrix multiplication between the object query and pixel features. The predicted masks are matched with ground-truth masks in a one-to-one manner through Hungarian matching [45] and are supervised accordingly. Moreover, as the number of object queries is often greater than the number of labeled masks, only a subset of predicted masks are optimized through this matching process. We apply no penalty to the remaining unmatched proposals, which ensures that more mask proposals are obtained.
In-Vocabulary Classifier Once the mask proposals are predicted, they are classified with category text embedding in a contrastive manner, where the class embeddings for each mask and category text embeddings are projected into a common embedding space. That is, the predicted class probability by in-vocabulary classifier is defined as follows:  $\forall i = 1,\dots ,N$
$$
\hat {c} _ {i, i n} = \operatorname {s o f t m a x} \left(\frac {1}{T} \left[ \cos \left(\mathbf {v} _ {i}, \mathbf {t} _ {1}\right), \cos \left(\mathbf {v} _ {i}, \mathbf {t} _ {2}\right), \dots , \cos \left(\mathbf {v} _ {i}, \mathbf {t} _ {| C |}\right) \right]\right), \tag {6}
$$
where  $T$  is a learnable temperature parameter with initialization of 0.07 to control the sharpness of the distribution,  $\cos$  is cosine distance measurement,  $\mathbf{v}_i$  is the class embeddings for  $i$ -th predicted mask, which is obtained by mask pooling over the final pixel features from pixel decoder, similar to [29].  $\mathbf{t}_j$  is the category name's text embeddings of class  $j$ , which is obtained by feeding the category name to a CLIP-pretrained text encoder. Note that these category text embeddings only need to be generated once. They are then kept in memory to serve as text classifiers, and thus it incurs negligible additional cost during training. This forms our in-vocabulary classifier.
Out-of-Vocabulary Classifier During inference, however, we notice that using the in-vocabulary classifier alone fails to generalize to completely novel unseen classes, as the model is only trained on a finite set of categories and thus could not recognize diverse novel concepts. To address this issue, we introduce an out-of-vocabulary classifier, which applies mask pooling to the frozen CLIP backbone features, aiming to borrow the pretrained (intact) open-vocabulary recognition ability from CLIP. Unlike the other two-stage methods [90, 52, 25, 89], where one or multiple forward processes of CLIP are needed, the adopted out-of-vocabulary classifier introduces marginal additional costs, since the backbone features are already extracted (and only lightweight mask-pooling is performed). The predicted class probability by out-of-vocabulary classifier  $\hat{c}_{i,out}$  is then obtained in a manner similar to Eq. (6) by replacing  $\mathbf{v}_i$  with the mask-pooled features over frozen CLIP backbone features. This classifier strictly maintains the original CLIP feature distribution, allowing us to better recognize brand new categories. Note that the out-of-vocabulary classifier is only performed during testing.
Combining In- and Out-of-Vocabulary Classifiers Following prior works [31, 29, 46, 89], we employ geometric ensemble to fuse the classification scores between in-vocabulary and out-of-vocabulary classifiers. That is,  $\forall j = 1,\dots ,|C|$
$$
\hat {c} _ {i} (j) = \left\{ \begin{array}{l l} \left(\hat {c} _ {i, i n} (j)\right) ^ {(1 - \alpha)} \cdot \left(\hat {c} _ {i, o u t} (j)\right) ^ {\alpha}, & \text {i f} j \in C _ {t r a i n} \\ \left(\hat {c} _ {i, i n} (j)\right) ^ {(1 - \beta)} \cdot \left(\hat {c} _ {i, o u t} (j)\right) ^ {\beta}, & \text {o t h e r w i s e} \end{array} \right. \tag {7}
$$
where  $\hat{c}_i(j)$  denotes the  $j$ -th element of  $\hat{c}_i$ , and the superscripts in and out refer to in-vocabulary and out-of-vocabulary classifier, respectively.  $\alpha, \beta \in [0,1]$  balance the predictions between in- and out-of-vocabulary classifiers for seen and novel unseen categories.
Herein, we provide implementation details of FC-CLIP in Sec. 4.1. After setting the stage, we introduce our main results, compared with state-of-the-art methods and ablations studies in Sec. 4.2.
Architecture We use ConvNeXt-Large CLIP [58, 69] backbones from OpenCLIP  $[38]^1$  pretrained on LAION-2B [74] dataset. On top of the CLIP backbone, we build the mask generator, following Mask2Former [20]. Nine mask decoders are employed to generate the class-agnostic masks by taking as inputs the enhanced pixel features and a set of object queries. For in-vocabulary classification, following [29], the class embeddings are obtained by mask-pooling the pixel features from the pixel decoder's final output. Afterwards, the classification logits (before softmax) is obtained by matrix multiplication between the predicted class embeddings and categories' text embeddings.
Training Strategy We follow [20] and adopt the same training recipe and losses without any special design. The training is optimized with AdamW [41, 61] optimizer and weight decay 0.05. We use a crop size of  $1024 \times 1024$ . We employ the learning rate  $1 \times 10^{-4}$  and a multi-step decay schedule. The training batch size is 16, and the model is trained for 50 epochs on COCO panoptic training set [54].
Inference Strategy During inference, the shorted side of input images will be resized to 800 while ensuring longer side not exceeds 1333. For Cityscapes and Mapillary Vistas, we increase the shorter side size to 1024. We adopt mask-wise merging scheme [20] for the mask predictions. The out-of-vocabulary classifier is only performed during inference by mask-pooling over the frozen CLIP backbone features. The final classification results are then obtained by geometric ensembling in- and out-of-vocabulary classifiers [31, 29, 46, 89], as in Eq. (7), where we default  $\alpha = 0.4$  and
<https://github.com/mlfoundations/open Clip>
Table 1: Open-vocabulary panoptic segmentation performance on ADE20K. The proposed FCCLIP demonstrates better performances than prior arts, while using much fewer frozen parameters. We provide more results in the supplementary material
method	params (M)	zero-shot test dataset ADE20K	training dataset COCO
frozen	trainable	PQ	AP	mIoU	PQ	AP	mIoU
MaskCLIP [25]	304	63	15.1	6.0	23.7	-	-	-
FreeSeg [68]	-	-	16.3	6.5	24.6	-	-	-
ODISE [89]	1494	28	22.6	14.4	29.9	55.4	46.0	65.2
ODISE [89] (caption)	1494	28	23.4	13.9	28.7	45.6	38.4	52.4
FC-CLIP (ours)	200	21	26.8	16.8	34.1	54.4	44.6	63.7
Table 2: Open-vocabulary panoptic segmentation performance on street-view datasets. The proposed FC-CLIP demonstrates better transferability to street-view dataset
method	zero-shot test dataset
Mapillary Vistas	Cityscapes
PQ	SQ	RQ	mIoU	PQ	SQ	RQ	AP	mIoU
ODISE [89]	14.2	61.0	17.2	-	23.9	75.3	29.0	-	-
FC-CLIP (ours)	18.2	57.7	22.9	27.9	44.0	75.4	53.6	26.8	56.2
$\beta = 0.8$ . Following prior arts, we also adopt prompt engineering from [29, 89] and prompt templates from [31, 52]. If not specified, FC-CLIP is only trained on COCO panoptic dataset [54]. Following prior works [29, 89], we zero-shot evaluate the model on ADE20K [100], Cityscapes [22], and Mapillary Vistas [64] for open-vocabulary panoptic segmentation. We also report open-vocabulary semantic segmentation results on those datasets along with PASCAL datasets [27, 63]. The panoptic segmentation results are evaluated with the panoptic quality (PQ) [44], Average Precision (AP), and mean intersection-over-union (mIoU), and semantic segmentation is evaluated with mIoU [27]. Note that all results are obtained with the same single checkpoint trained on COCO panoptic data only.
We summarize the main results for open-vocabulary panoptic segmentation and semantic segmentation in Tab. 1, Tab. 2 and Tab. 3, where we train FC-CLIP on COCO train set with panoptic annotation and evaluate it on various datasets in a zero-shot manner.
Open-Vocabulary Panoptic Segmentation Evaluation on ADE20K In Tab. 1, we compare our FC-CLIP with other state-of-the-art methods on ADE20K [100], the main test-bed of zero-shot open-vocabulary panoptic segmentation. As shown in the table, our method achieves significantly better performance compared to MaskCLIP [25], with  $+11.7\mathrm{PQ}$ ,  $+10.8\mathrm{AP}$  and  $+10.4\mathrm{mIoU}$ , even though we use fewer frozen  $(-66\mathrm{M})$  and trainable  $(-42\mathrm{M})$  parameters. When compared to the concurrent methods FreeSeg [68] and ODISE [89], the advantage of FC-CLIP persists. FC-CLIP is  $+10.5\mathrm{PQ}$ ,  $+10.3\mathrm{AP}$ , and  $+9.5\mathrm{mIoU}$  better than FreeSeg without using COCO-Stuff annotations [5] (which contains more semantic classes than COCO-Panoptic). Our PQ, AP, mIoU score are also  $+4.2$ ,  $+2.4$ ,  $+4.2$  higher than ODISE under the same training settings. Compared to ODISE with caption [15] for supervision, our model still outperforms it by  $+3.4\mathrm{PQ}$ , setting a new state-of-the-art record. Meanwhile, it is noticeable that our model has  $6.3\times (5.9\times)$  significantly fewer frozen (total) parameters compared to ODISE, which utilizes a strong large backbone from stable diffusion [71] for feature extraction.
Open-Vocabulary Panoptic Segmentation Evaluation on Street-View Datasets In Tab. 2, we evaluate on Cityscapes and Mapillary Vistas, which focus on street driving scenes. Compared to state-of-the-art method ODISE, FC-CLIP achieves better performances on both datasets. Specifically, it outperforms ODISE by  $+4.0$  PQ and  $+20.1$  PQ on Mapillary Vistas and Cityscapes, respectively. Notably, FC-CLIP has a slightly lower SQ, which indicates our mask generator is actually weaker than the one in ODISE, which utilizes a much larger backbone.
Open-Vocabulary Semantic Segmentation Evaluation Although our model was trained on COCO panoptic data only, it also performs well on open-vocabulary semantic segmentation. In Tab. 3, we
Table 3: Open-vocabulary semantic segmentation performance. The proposed FC-CLIP also demonstrates state-of-the-art performances on open-vocabulary semantic segmentation
method	training dataset	mIoU
A-847	PC-459	A-150	PC-59	PAS-21	PAS-20
SPNet [85]	Pascal VOC [27]	-	-	-	24.3	18.3	-
ZS3Net [4]	Pascal VOC [27]	-	-	-	19.4	38.3	-
LSeg [48]	Pascal VOC [27]	-	-	-	-	47.4	-
GroupViT [88]	GCC [75]+YFCC [79]	4.3	4.9	10.6	25.9	50.7	52.3
SimBaseline [90]	COCO Stuff [5]	-	-	15.3	-	74.5	-
ZegFormer [24]	COCO Stuff [5]	-	-	16.4	-	73.3	-
LSeg+ [48, 29]	COCO Stuff [5]	3.8	7.8	18.0	46.5	-	-
OVSeg [52]	COCO Stuff [5]	9.0	12.4	29.6	55.7	-	94.5
SAN [91]	COCO Stuff [5]	13.7	17.1	33.3	60.2	-	95.5
OpenSeg [29]	COCO Panoptic + COCO Caption	6.3	9.0	21.1	42.1	-	-
ODISE [89] (caption)	COCO Panoptic + COCO Caption	11.0	13.8	28.7	55.3	82.7	-
MaskCLIP [25]	COCO Panoptic	8.2	10.0	23.7	45.9	-	-
ODISE [89]	COCO Panoptic	11.1	14.5	29.9	57.3	84.6	-
FC-CLIP (ours)	COCO Panoptic	14.8	18.2	34.1	58.4	81.8	95.4
Table 4: FPS comparison. All results are obtained with one V100 GPU, CUDA 11.6 and PyTorch 1.13, by taking the average runtime on the entire validation set, including post-processing time
method	ADE20K	COCO
ODISE [89]	0.41	0.39
FC-CLIP (ours)	2.71 (6.61×)	2.76 (7.08×)
report our model's performance on various benchmarks against other open-vocabulary segmentation models, where FC-CLIP shows an overall superior performance. Specifically, with the same training annotations used, FC-CLIP outperforms MaskCLIP by  $+6.6$ ,  $+8.2$ ,  $+10.4$ ,  $+12.5$  mIoU across A-847, PC-459, A-150, and PC-59, respectively. Compared to methods with caption annotations, FC-CLIP persists its advantages, where it outperforms ODISE (caption) by  $+3.8$ ,  $+4.4$ ,  $+5.4$ ,  $+3.1$  mIoU across datasets A-847, PC-459, A-150, PC-59 respectively. Against other open-vocabulary semantic segmentation methods, our model maintains its advantages across different datasets, despite being trained solely with panoptic annotations. Furthermore, it demonstrates comparable performance to state-of-the-art open-vocabulary semantic segmentation methods, which utilize the COCO-Stuff dataset as their training set. The COCO-Stuff dataset comprises 171 classes, 38 more classes than COCO-Panoptic, and offers highly desirable annotations for semantic segmentation tasks. It is worth mentioning that these methods build their approach on top of ViT-L (with extra designs [91]), resulting in a significantly larger model size compared to our deployed ConvNeXt-L (304M vs. 198M). Despite the disparity in model size, FC-CLIP remains competitive in terms of performance. Specifically, FC-CLIP outperforms state-of-the-art open-vocabulary semantic segmentation method SAN [91] by 1.1 and 1.1 mIoU on the challenging A-847 and PC-459 datasets.
Inference Speed We provide a comparison of FPS (frames per second) in Tab. 4. The proposed FC-CLIP not only demonstrates superior performances, but also enjoys a significant fast inference time: FC-CLIP runs  $6.61 \times$  and  $7.08 \times$  faster than ODISE evaluated on ADE20K and COCO datasets, respectively.
Training on ADE20K and Evaluating on COCO We further validate the effectiveness of FC-CLIP by using a different training dataset. Specifically, we follow [68, 89] to train our model on ADE20K dataset with panoptic annotation, and evaluate it on COCO panoptic dataset. As shown in Tab. 5, FC-CLIP outperforms FreeSeg [68] by  $+10.5$  PQ, and ODISE [89] by  $+2.0$  PQ on COCO dataset. Notably, our model actually has a lower SQ  $(-1.4)$  compared to ODISE, which utilizes a much larger backbone and thus has a stronger mask generator. Nevertheless, FC-CLIP still outperforms ODISE significantly with a simple yet effective design.
Fine-tuning CLIP Backbone Harms Performance on Novel Vocabularies We validate the necessity of freezing CLIP backbone to ensure a better generalization to novel vocabularies. We compare the performance of trainable CLIP variant and frozen CLIP variant in Fig. 4, where we use the same mask proposals to ensure a fair comparison. Specifically, we compare the performance on
Table 5: Results of training on ADE20K panoptic and evaluating on COCO panoptic val set. The proposed FC-CLIP performs better than prior arts, even in the different setting (i.e., trained on ADE20K and zero-shot evaluated on COCO)
method	zero-shot test dataset	training dataset
COCO	ADE20K
PQ	SQ	RQ	PQ	SQ	RQ
FreeSeg [68]	16.5	72.0	21.6	-	-	-
ODISE [89]	25.0	79.4	30.4	31.4	77.9	36.9
FC-CLIP (ours)	27.0	78.0	32.9	41.9	78.2	50.2
Figure 4: Trainable CLIP vs. Frozen CLIP, with per-class PQ analysis. We show 10 common classes (labeled in green) shared by COCO and ADE20K, and 10 novel classes (labeled in red) that are only in ADE20K. The frozen CLIP demonstrates a much better recognition ability for novel classes, while performing similarly for the seen classes.
10 seen classes, which are shared by both COCO and ADE20K (e.g., person, sky), and 10 unseen classes, which are only included in ADE20K dataset (e.g., arcade machine, dishwasher). As shown in the figure, tuning CLIP backbone leads to a worse performance on unseen concepts, which breaks the CLIP feature alignment and thus loses its recognition ability on a much wider vocabulary.
In this work, we have presented FC-CLIP, a simple yet effective single-stage framework for open-vocabulary segmentation. FC-CLIP shows great potential by building everything on top of a shared frozen convolutional CLIP backbone, which not only significantly reduces training and testing costs, but also establishes a strong baseline on multiple benchmarks. Our study demonstrates how to better adapt a pretrained CLIP model for downstream dense prediction tasks, which we hope will shed the light on unleashing CLIP's potential for other various downstream tasks.
Limitations FC-CLIP presents a simple single-stage open-vocabulary segmentation framework with state-of-the-art performance. We note that there exist some interesting research topics to be explored in the near future, such as better unleashing CLIP's potential in both mask segmentation and classification, how to deal with conflict or overlapping vocabularies (e.g., cat vs. cat head), etc.
Broader Impact FC-CLIP shows great potential for segmenting and naming every object in the scene, which could facilitate many applications including intelligent home assistants, robots, self-driving, etc. Yet it relies on CLIP model pre-trained on the Internet data that may be biased, which calls for future research for calibration to avoid misuse.
[87] Yuwen Xiong, Renjie Liao, Hengshuang Zhao, Rui Hu, Min Bai, Ersin Yumer, and Raquel Urtasun. Upsnet: A unified panoptic segmentation network. In CVPR, 2019.
[88] Jiarui Xu, Shalini De Mello, Sifei Liu, Wonmin Byeon, Thomas Breuel, Jan Kautz, and Xiaolong Wang. Groupvit: Semantic segmentation emerges from text supervision. In CVPR, 2022.
[89] Jiarui Xu, Sifei Liu, Arash Vahdat, Wonmin Byeon, Xiaolong Wang, and Shalini De Mello. Open-vocabulary panoptic segmentation with text-to-image diffusion models. In CVPR, 2023.
[90] Mengde Xu, Zheng Zhang, Fangyun Wei, Yutong Lin, Yue Cao, Han Hu, and Xiang Bai. A simple baseline for zero-shot semantic segmentation with pre-trained vision-language model. In ECCV, 2022.
[91] Mengde Xu, Zheng Zhang, Fangyun Wei, Han Hu, and Xiang Bai. Side adapter network for open-vocabulary semantic segmentation. In CVPR, 2023.
[92] Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. Coca: Contrastive captioners are image-text foundation models. TMLR, 2022.
[93] Qihang Yu, Huiyu Wang, Dahun Kim, Siyuan Qiao, Maxwell Collins, Yukun Zhu, Hartwig Adam, Alan Yuille, and Liang-Chieh Chen. Cmt-deeplab: Clustering mask transformers for panoptic segmentation. In CVPR, 2022.
[94] Qihang Yu, Huiyu Wang, Siyuan Qiao, Maxwell Collins, Yukun Zhu, Hartwig Adam, Alan Yuille, and Liang-Chieh Chen. k-means Mask Transformer. In ECCV, 2022.
[95] Lu Yuan, Dongdong Chen, Yi-Ling Chen, Noel Codella, Xiyang Dai, Jianfeng Gao, Houdong Hu, Xuedong Huang, Boxin Li, Chunyuan Li, et al. Florence: A new foundation model for computer vision. arXiv:2111.11432, 2021.
[96] Yuhui Yuan, Xilin Chen, and Jingdong Wang. Object-contextual representations for semantic segmentation. In ECCV, 2020.
[97] Alireza Zareian, Kevin Dela Rosa, Derek Hao Hu, and Shih-Fu Chang. Open-vocabulary object detection using captions. In CVPR, 2021.
[98] Pengchuan Zhang, Xiujun Li, Xiaowei Hu, Jianwei Yang, Lei Zhang, Lijuan Wang, Yejin Choi, and Jianfeng Gao. Vinvl: Revisiting visual representations in vision-language models. In CVPR, 2021.
[99] Sixiao Zheng, Jiachen Lu, Hengshuang Zhao, Xiatian Zhu, Zekun Luo, Yabiao Wang, Yanwei Fu, Jianfeng Feng, Tao Xiang, Philip HS Torr, et al. Rethinking semantic segmentation from a sequence-to-sequence perspective with transformers. In CVPR, 2021.
[100] Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In CVPR, 2017.
[101] Chong Zhou, Chen Change Loy, and Bo Dai. Extract free dense labels from clip. In ECCV, 2022.
[102] Ziqin Zhou, Yinjie Lei, Bowen Zhang, Lingqiao Liu, and Yifan Liu. Zegclip: Towards adapting clip for zero-shot semantic segmentation. In CVPR, 2023.
[103] Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable detr: Deformable transformers for end-to-end object detection. In ICLR, 2021.
[104] Xueyan Zou, Zi-Yi Dou, Jianwei Yang, Zhe Gan, Linjie Li, Chunyuan Li, Xiyang Dai, Jianfeng Wang, Lu Yuan, Nanyun Peng, Lijuan Wang, Yong Jae Lee, and Jianfeng Gao. Generalized decoding for pixel, image, and language. In CVPR, 2023.
Appendix In the following supplementary materials, we present additional experimental results pertaining to the design of FC-CLIP. Our supplementary analysis also includes comparisons against other methods that specifically address open-vocabulary semantic segmentation, ensemble methods, and hyperparameter tuning. Furthermore, we provide a quantitative comparison between ViT-based CLIP and CNN-based CLIP across varying input sizes, along with additional visualizations and comprehensive dataset details.
6 Additional Experimental Results
Fine-tuning or Freezing CLIP Backbone in FC-CLIP In this study, we provide a comprehensive analysis of the impact of fine-tuning or freezing the CLIP backbone in our framework. We specifically focus on the PQ $^{\text{seen}}$  and PQ $^{\text{unseen}}$  metrics, which evaluate the performance for classes that overlap and do not overlap between the training and testing datasets, respectively. To determine whether a class is seen or unseen, we adopt the prompt engineering technique described in [29], which provides synonyms or subcategories of classes. Specifically, if any category name in test dataset overlaps with a category name in training dataset, we consider it as a seen class; otherwise unseen. As discussed in the main paper, the proposed FC-CLIP contains three components: a class-agnostic mask generator, an in-vocabulary classifier, and an out-of-vocabulary classifier. We thus explore using frozen or trainable CLIP for each component, and summarize the results in Tab. 6. To ensure a fair comparison, all "trainable" modules utilize the same weights, resulting in identical mask proposals and in-vocabulary classification results. Moreover, we note that the first row in Tab. 6 with trainable mask generator and in-vocabulary classifier, can be considered as an approximation to OpenSeg [29] in our framework. Our findings reveal that an in-vocabulary classifier built upon a trainable CLIP backbone achieves a higher PQ $^{\text{seen}}$  score (37.9 compared to 32.4), but experiences a decrease in PQ $^{\text{seen}}$  (2.6 compared to 12.6) compared to a frozen out-of-vocabulary classifier. Consequently, a model that incorporates a trainable CLIP backbone for all components yields a PQ of 24.1, which is 2.7 lower than our final model (last row) that relies on a single frozen CLIP backbone. Using a trainable mask generator and in-vocabulary classifier, along with a frozen out-of-vocabulary classifier boosts the performance but requires maintaining one trainable and one frozen CLIP weights, resulting in  $2 \times$  more backbone parameters. In summary, our observations demonstrate that building the entire framework upon a frozen CLIP backbone is not only effective but also efficient, providing a better balance between PQ $^{\text{seen}}$  and PQ $^{\text{unseen}}$  metrics.
Table 6: Effects of fine-tuning or freezing the CLIP backbone for each module in FC-CLIP. Building all three modules upon a single frozen CLIP backbone attains best performance. Note that our mask generator and in-vocabulary classifier use the same backbone following [20, 29, 94], and thus it is infeasible (denoted as N/A) for the setting in the 2nd last row. Our final setting is labeled in gray
mask generator	in-vocabulary classifier	out-of-vocabulary classifier	PQ	PQseen	PQunseen
trainable	trainable	-	17.7	37.9	2.6
trainable	-	frozen	21.1	32.4	12.6
trainable	trainable	trainable	24.1	38.9	13.1
trainable	trainable	frozen	25.4	40.0	14.6
trainable	frozen	frozen	N/A	N/A	N/A
frozen	frozen	frozen	26.8	39.5	17.3