DOI: 10.26188/28561820
title: <b>BiXT model weights</b> (Perceiving Longer Sequences with Bi-Directional Cross-Attention Transformers)

description: <b>BiXT model weights</b>This collection includes PyTorch weights of various BiXT models trained on the ImageNet dataset, as introduced in the paper: <b>Markus Hiller, Krista A. Ehinger, and Tom Drummond. "Perceiving Longer Sequences With Bi-Directional Cross-Attention Transformers."</b><b> </b><b><i>The Thirty-eighth Annual Conference on Neural Information Processing Systems (NeurIPS), 2024</i></b><b>. </b>(available here)These weights are in support of the following github repository containing all code: https://github.com/mrkshllr/BiXT/ In detail, weights for the following models are available: <b>ImageNet Classification Models</b><b>Default BiXT-Tiny models with 64 latents:</b>BiXT-Ti/16: <i>bixt_ti_l64_p16.zip</i>BiXT-Ti/8: <i>bixt_ti_l64_p16s8.zip</i>BiXT-Ti/4: <i>bixt_ti_l64_p16s4.zip</i><b>Previous BiXT-Tiny models fine-tuned on larger 384x384 images:</b>BiXT-Ti/16-ft384: <i>bixt_ti_l64_p16_ft384.zip</i>BiXT-Ti/8-ft384: <i>bixt_ti_l64_p16s8</i><i>_ft384</i><i>.zip</i>BiXT-Ti/4-ft384: <i>bixt_ti_l64_p16s4</i><i>_ft384</i><i>.zip</i><b>Convolutional Alternative: BiXT-Tiny w/ conv-tokeniser:</b>BiXT-Ti/16 (conv): <i>bixt_conv_ti_l64_p16.zip</i><b>Slightly larger models with embedding dimension of 256 instead of 192 (default tiny):</b>BiXT-d256/16: <i>bixt_ed256_l64_p16.zip</i>BiXT-d256/8: <i>bixt_ed256LS_l64_p16s8.zip</i>BiXT-d256/8-ft384: <i>bixt_ed256LS_l64_p16s8_ft384.zip</i><br><b>Models for Dense Downstream Tasks</b>Note that for standard ImageNet training, we simply use a standard classification loss on the average-pooled latent embeddings for training. This means that for a 12 layer BiXT network, the refined patch tokens only receive a gradient until layer 11 -- which is why we employ only a <i>one-sided cross-attention</i> for the last layer (see BiXT model file here).For simplicity and easy transfer to dense downstream tasks, we therefore simply create and train BiXT-models with a depth of 13 and train these on ImageNet (see here); Afterwards, the last one-sided cross-attention that exclusively refines the latent vectors is simply discarded and the remaining (fully-trained) 12-layer network is used for finetuning on downstream tasks.Note: It is, of course, entirely possible to replace or extend our simple classification loss on the averaged latent vectors through other token-side losses (e.g. Masked Image Modelling) to provide a gradient signal for the token side and thereby directly train both, the latent and token refinement for all layers.<b>Dense (d13) BiXT-Tiny models with 64 latents:</b>BiXT-Ti/16 (d13): <i>bixt_ti_l64_d13_p16.zip</i>BiXT-Ti/8 (d13): <i>bixt_ti_l64</i><i>_d13</i><i>_p16s8.zip</i>BiXT-Ti/4 (d13): <i>bixt_ti_l64</i><i>_d13</i><i>_p16s4.zip</i><b>Previous d</b><b>ense (d13) </b><b>BiXT-Tiny models fine-tuned on larger 384x384 images:</b>BiXT-Ti/16-ft384 (d13): <i>bixt_ti_l64</i><i>_d13</i><i>_p16_ft384.zip</i>BiXT-Ti/8-ft384 (d13): <i>bixt_ti_l64</i><i>_d13</i><i>_p16s8</i><i>_ft384</i><i>.zip</i><b>Dense Convolutional Alternative: BiXT-Tiny</b><b> (d13)</b><b> w/ conv-tokeniser:</b>BiXT-Ti/16 (conv, d13): <i>bixt_conv_ti_l64_d13_p16.zip</i>BiXT-Ti/8 (conv, d13): <i>bixt_conv_ti_l64_d13_p8.zip</i><br><b>Note</b>In case you need any further information or model files, please make sure to check out the code in the github repository here and/or reach out! Additional information as well as details on the performance of each pre-trained model is also provided in the repository.