Package: tfswin
tfswinKeras (TensorFlow v2) reimplementation ofSwin TransformerandSwin Transformer V2models.Based onOfficial Pytorch implementation.Supports variable-shape inference for downstream tasks.Contains pretrained weights converted from official ones.InstallationpipinstalltfswinExamplesDefault usage (without preprocessing):fromtfswinimportSwinTransformerTiny224# + 5 other variants and input preprocessing# or# from tfswin import SwinTransformerV2Tiny256  # + 5 other variants and input preprocessingmodel=SwinTransformerTiny224()# by default will download imagenet[21k]-pretrained weightsmodel.compile(...)model.fit(...)Custom classification (with preprocessing):fromkerasimportlayers,modelsfromtfswinimportSwinTransformerTiny224,preprocess_inputinputs=layers.Input(shape=(224,224,3),dtype='uint8')outputs=layers.Lambda(preprocess_input)(inputs)outputs=SwinTransformerTiny224(include_top=False)(outputs)outputs=layers.Dense(100,activation='softmax')(outputs)model=models.Model(inputs=inputs,outputs=outputs)model.compile(...)model.fit(...)DifferencesCode simplification:Pretrain input height and width are always equalPatch height and width are always equalAll input shapes automatically evaluated (not passed through a constructor like in PyTorch)Downsampling have been moved out from basic layer to simplify feature extraction in downstream tasks.Performance improvements:Layer normalization epsilon fixed at1.001e-5, inputs are casted tofloat32to use fused op implementation.Some layers have been refactored to use faster TF operations.A lot of reshapes have been removed. Most of the time internal representation is 4D-tensor.Attention mask and relative index estimations moved to basic layer level.Variable shapesSwin Transformer receptive field is larger or equal to pretraining image size. Window reduction is used in image
classificationV1andV2pipelines. E.g.:SwinTransformerTiny224: last stage size is 7x7 with window size 7, no shift for last stage.SwinTransformerLarge384: last stage size is 12x12 with window size 12, no shift for last stage.SwinTransformerV2Tiny256: last stages sizes are 16x16 and 8x8 with window size 16 and 16->6, no shift for 2 last
stages.SwinTransformerV2Large384: last stages sizes are 24x24 and 12x12 with window size 24 and 24->12, no shift for 2 last
stages.But there is no such trick in semantic segmentationbackbone.
This reimplementation always applies window reduction conditioned on dynamic input height and width.When using Swin models with input shapes different from pretraining one, try to make height and width to be multiple
of32 * window_size. Otherwise a lot of tensors will be padded, resulting in speed degradation.EvaluationFor correctness,TinyandSmallmodels (original and ported) tested
withImageNet-v2 test set.Note: Swin models are very sensitive to input preprocessing (bicubic resize in the original evaluation script).importtensorflowastfimporttensorflow_datasetsastfdsfromtfswinimportSwinTransformerTiny224,preprocess_inputdef_prepare(example,input_size=224,crop_pct=0.875):scale_size=tf.math.floor(input_size/crop_pct)image=example['image']shape=tf.shape(image)[:2]shape=tf.cast(shape,'float32')shape*=scale_size/tf.reduce_min(shape)shape=tf.round(shape)shape=tf.cast(shape,'int32')image=tf.image.resize(image,shape,method=tf.image.ResizeMethod.BICUBIC)image=tf.round(image)image=tf.clip_by_value(image,0.,255.)image=tf.cast(image,'uint8')pad_h,pad_w=tf.unstack((shape-input_size)//2)image=image[pad_h:pad_h+input_size,pad_w:pad_w+input_size]image=preprocess_input(image)returnimage,example['label']imagenet2=tfds.load('imagenet_v2',split='test',shuffle_files=True)imagenet2=imagenet2.map(_prepare,num_parallel_calls=tf.data.AUTOTUNE)imagenet2=imagenet2.batch(8)model=SwinTransformerTiny224()model.compile('sgd','sparse_categorical_crossentropy',['accuracy','sparse_top_k_categorical_accuracy'])history=model.evaluate(imagenet2)print(history)nameoriginal acc@1ported acc@1original acc@5ported acc@5Swin-T V167.6467.8187.8487.87Swin-S V170.6670.8089.3489.49Swin-T V271.6972.0090.0490.06Swin-S V273.2073.5791.2491.11Note: Swin V1 model were evaluated with wrong preprocessing (distorted aspect ratio) and ImageNet-1K weights which were
replaced with ImageNet-21K weights in 3.0.0 release.The most metric differences comes from input data preprocessing (decoding, interpolation).
All layers outputs have been compared with original ones.
Most of them have maximum absolute difference around9.9e-5.
Maximum absolute difference among all layers is3.5e-4.Citation@inproceedings{liu2021Swin,
  title={Swin Transformer: Hierarchical Vision Transformer using Shifted Windows},
  author={Liu, Ze and Lin, Yutong and Cao, Yue and Hu, Han and Wei, Yixuan and Zhang, Zheng and Lin, Stephen and Guo, Baining},
  booktitle={Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)},
  year={2021}
}@inproceedings{liu2021swinv2,
  title={Swin Transformer V2: Scaling Up Capacity and Resolution}, 
  author={Ze Liu and Han Hu and Yutong Lin and Zhuliang Yao and Zhenda Xie and Yixuan Wei and Jia Ning and Yue Cao and Zheng Zhang and Li Dong and Furu Wei and Baining Guo},
  booktitle={International Conference on Computer Vision and Pattern Recognition (CVPR)},
  year={2022}
}