Question: What are the 24 different structures generated by random permutation in section 4?
Evidence: Following the scaling approach described in Section 3.3, we set 2× FLOPs of the initial or selected model from the previous step as the target hardware-cost in each step when individually scaling each factor, as summarized in Figure. All networks are trained for 300 epochs on ImageNet using the same training recipe with the one in DeiT, more details are included in Appendix E. We summarize our observations as follow: Scaled ViT models outperform SOTA DeiT models. As shown in Table, our scaled ViT models (e.g., DeiT-Scaled-Tiny/Small/Base) achieve a ↑0.4% ∼ ↑1.9% higher top-1 accuracy on ImageNet under the same FLOPs constraints. Specifically, our DeiT-Scaled-Tiny model chooses to use a smaller image resolution (i.e., 160×160 vs. 224×224) and more layers and a higher number of heads as compared to the SOTA DeiT-Tiny model, and thus achieves a ↑1.9% higher accuracy at the same cost in terms of FLOPs, while our DeiT-Scaled-Small/Base models choose to use a larger image resolution (i.e., 320/256×320/256 vs. 224×224) and more layers, together with a lower number of heads as compared to the SOTA DeiT-Small/Base model, helping them to achieve a ↑0.4% higher accuracy under similar FLOPs. This set of experiments shows that our simple search method can (1) effectively locate ViT models with better accuracy-FLOPs trade-offs and (2) automatically adapt different scaling factors towards the optimal accuracy-FLOPs trade-offs, e.g., different model shapes and structures at different scales of FLOPs.
Random permutation further boosts the performance. Inspired by the coarse-to-fine architecture selection scheme adopted in , we further randomly permute the scaling factors (i.e., d, h, e, r, I, and p) of each scaled model in Table. After the permutation, we select 24 architectures under the same target hardware-cost with the scaled model by iterative greedy search for each scaled model. Figure demonstrates that () such a random permutation can slightly push forward the frontier of accuracy-FLOPs trade-off (e.g., a ↑0.4% higher accuracy under similar FLOPs on top of the scaled models resulting from the adopted simple scaling method); and (2) our adopted iterative greedy search alone is sufficiently effective while requiring a lower exploration cost (e.g., 6 vs. 30 (6+24) models to be trained for each step as compared to such a search method together with the aforementioned permutation).  Scaled ViT also benefits from a longer training time. As pointed out by, training ViT models for more epochs (e.g., 1000 epochs) can further improve the achieved accuracy.
DeiT
To verify whether the scaled ViT models can benefit from more training epochs, we train the models in Table for 1000 epochs following the training recipe in. As shown in Table, longer training epochs also help our scaled models (e.g., DeiT-Scaled-Tiny/Small) to achieve a higher accuracy, and thus, the advantage of our scaled models over DeiT is consistent under both the 300-epochs training recipe and 1000-epochs training recipe, e.g., a ↑1.9% higher accuracy over DeiT-Tiny with 300 epochs vs. a ↑1.7% higher accuracy over DeiT-Tiny with 1000 epochs.
Drawn insights from scaling ViT. Based on the observations from the above experiments, especially the scaling strategies illustrated in Figure, we draw the following scaling insights dedicated to ViT:
(1) When targeting relatively small models (i.e., with smaller FLOPs than DeiT-Scaled-Small), the optimal models tend to select "scaling h (i.e., the number of heads)" or "scaling d (i.e., the number of layers)" and a "smaller I (i.e., the input image resolution)" (e.g., 160 × 160 instead of the commonly used 224 × 224).
FLOPs (G) ImageNet top-1 accuracy (%)
Figure: Resulting models from our iterative greedy search, where models achieving the best accuracy-FLOPs trade-offs are marked as DeiT-Scaled-Tiny/Small/Base. The architecture configurations (i.e., sets of d, h, e, r, I, and p) leading to these best models are extracted as our scaling strategies dedicated to ViT.
(2) When targeting relatively large models (i.e., with larger FLOPs than DeiT-Scaled-Small), the optimal models mainly select to "scaling I (i.e., the input image resolution)", while "slowing down scaling h (i.e., number of heads)" as compared to the case when targeting relatively small models.
After the permutation, we select 24 architectures under the same target hardware-cost with the scaled model by iterative greedy search for each scaled model.
Answer: The random permutation is implemented by randomly mutating the scaling factors (i.e., the number of layers, number of heads, embedding dimension per head, linear projection ratio, image resolution, and patch size) of the scaled architectures that are obtained by first performing iterative greedy search and then selecting those under the target hardware-cost. We have also added more details to the corresponding part for further clarification in Section 4.1.