Package: perceiver-model
Perceiver - PytorchImplementation ofPerceiver, General Perception with Iterative Attention, in Pytorch.
Extended from Phil Wang'sperceiver-pytorchYannic Kilcher explanation!Install$pipinstallperceiver-modelUsageimporttorchfromperceiver_pytorchimportPerceivermodel=Perceiver(input_channels=3,# number of channels for each token of the inputinput_axis=2,# number of axis for input data (2 for images, 3 for video)num_freq_bands=6,# number of freq bands, with original value (2 * K + 1)max_freq=10.,# maximum frequency, hyperparameter depending on how fine the data isdepth=6,# depth of net. The shape of the final attention mechanism will be:#   depth * (cross attention -> self_per_cross_attn * self attention)num_latents=256,# number of latents, or induced set points, or centroids. different papers giving it different nameslatent_dim=512,# latent dimensioncross_heads=1,# number of heads for cross attention. paper said 1latent_heads=8,# number of heads for latent self attention, 8cross_dim_head=64,# number of dimensions per cross attention headlatent_dim_head=64,# number of dimensions per latent self attention headnum_classes=1000,# output number of classesattn_dropout=0.,ff_dropout=0.,weight_tie_layers=False,# whether to weight tie layers (optional, as indicated in the diagram)fourier_encode_data=True,# whether to auto-fourier encode the data, using the input_axis given. defaults to True, but can be turned off if you are fourier encoding the data yourselfself_per_cross_attn=2# number of self attention blocks per cross attention)img=torch.randn(1,224,224,3)# 1 imagenet image, pixelizedmodel(img)# (1, 1000)For the backbone ofPerceiver IO, the follow up paper that allows for flexible number of output sequence length, just importPerceiverIOinsteadimporttorchfromperceiver_pytorchimportPerceiverIOmodel=PerceiverIO(dim=32,# dimension of sequence to be encodedqueries_dim=32,# dimension of decoder querieslogits_dim=100,# dimension of final logitsdepth=6,# depth of netnum_latents=256,# number of latents, or induced set points, or centroids. different papers giving it different nameslatent_dim=512,# latent dimensioncross_heads=1,# number of heads for cross attention. paper said 1latent_heads=8,# number of heads for latent self attention, 8cross_dim_head=64,# number of dimensions per cross attention headlatent_dim_head=64,# number of dimensions per latent self attention headweight_tie_layers=False# whether to weight tie layers (optional, as indicated in the diagram))seq=torch.randn(1,512,32)queries=torch.randn(1,128,32)logits=model(seq,queries=queries)# (1, 128, 100) - (batch, decoder seq, logits dim)Citations@misc{jaegle2021perceiver,title={Perceiver: General Perception with Iterative Attention},author={Andrew Jaegle and Felix Gimeno and Andrew Brock and Andrew Zisserman and Oriol Vinyals and Joao Carreira},year={2021},eprint={2103.03206},archivePrefix={arXiv},primaryClass={cs.CV}}@misc{jaegle2021perceiver,title={Perceiver IO: A General Architecture for Structured Inputs & Outputs},author={Andrew Jaegle and Sebastian Borgeaud and Jean-Baptiste Alayrac and Carl Doersch and Catalin Ionescu and David Ding and Skanda Koppula and Andrew Brock and Evan Shelhamer and Olivier Hénaff and Matthew M. Botvinick and Andrew Zisserman and Oriol Vinyals and João Carreira},year={2021},eprint={2107.14795},archivePrefix={arXiv},primaryClass={cs.LG}}