Package: perceiver-io
Perceiver, Perceiver IO and Perceiver ARThis repository is a PyTorch implementation of Perceiver, Perceiver IO and Perceiver AR, with PyTorch Lightning
interfaces for model training and Hugging Face 🤗 interfaces for inference.Perceiver: General Perception with Iterative Attention
       (paper,video)Perceiver IO: A General Architecture for Structured Inputs & Outputs
      (paper,blog post)General-purpose, long-context autoregressive modeling withPerceiver AR(paper,blog post)OverviewCore of theperceiver-iolibrary arebackend models, lightweight PyTorch implementations of Perceiver,
Perceiver IO and Perceiver AR. They can be wrapped intoPyTorch Lightningmodules for training (Lightning interface) and 🤗 modules for inference (Hugging Face interface). Seelibrary designfor details.The command line interface for training is implemented withLightning CLI.
Training datasets are 🤗datasetswrapped into PyTorch Lightning data modules.
For NLP tasks,perceiver-iosupports all 🤗fast tokenizersand the 🤗 Perceiver UTF-8 bytes tokenizer.DocumentationInstallationGetting startedLibrary designPretrained modelsTraining examplesInference examplesModel constructionBuilding blocksInstallationVia pippipinstallperceiver-io[text,vision,audio]From sourcesInstallation from sources requires aMinicondaand aPoetry(1.2.0 or higher) installation.Create and activate theperceiver-ioconda environment:condaenvcreate-fenvironment.yml
condaactivateperceiver-ioInstall main and test dependencies, including all extras:# Without dependencies required for examplespoetryinstall--all-extrasIf you want to run theexampleslocally, additionally use--with examples:poetryinstall--all-extras--withexamplesDocker imagedockerpullghcr.io/krasserm/perceiver-io:latestSeeDocker imagefor details.Getting startedInferenceOptical flowCompute the optical flow between consecutive frames of an input video and write the rendered results to an output
video:fromurllib.requestimporturlretrievefromtransformersimportpipelinefromperceiver.data.visionimportvideo_utilsfromperceiver.model.visionimportoptical_flow# register auto-classes and pipelineurlretrieve(url="https://martin-krasser.com/perceiver/flow/sintel_clip_cave_dragon_fight.mp4",filename="sintel_clip_cave_dragon_fight.mp4",)# Create optical flow pipelineoptical_flow_pipeline=pipeline("optical-flow",model="krasserm/perceiver-io-optical-flow",device="cuda:0")# load consecutive video frame pairsframe_pairs=video_utils.read_video_frame_pairs("sintel_clip_cave_dragon_fight.mp4")# create and render optical flow for all frame pairsoptical_flows=optical_flow_pipeline(frame_pairs,render=True,device="cuda:0")# create video with rendered optical flowsvideo_utils.write_video("sintel_clip_cave_dragon_fight_output.mp4",optical_flows,fps=24)Here is a side-by-side comparison of the input and output video:Symbolic audio generationCreate audio sequences by generating symbolic (MIDI) audio data and converting the
generated audio symbols into WAV output usingfluidsynth(Note:fluidsynth must be installed
in order for the following example to work):fromtransformersimportpipelinefrompretty_midiimportPrettyMIDIfromperceiver.model.audioimportsymbolic# auto-class registrationrepo_id="krasserm/perceiver-ar-sam-giant-midi"prompt=PrettyMIDI("prompt.mid")audio_generator=pipeline("symbolic-audio-generation",model=repo_id)output=audio_generator(prompt,max_new_tokens=64,num_latents=1,do_sample=True,top_p=0.95,temperature=1.0,render=True)withopen("generated_audio.wav","wb")asf:f.write(output["generated_audio_wav"])Examples of generated audio sequences are available on the 🤗hub.Seeinference examplesfor more examples.TrainingTrain a small Perceiver IO image classifier (907K parameters) on MNIST from the command line. The classifier
cross-attends to individual pixels of input images withrepeated cross-attention.
Seeimage classificationtraining example for more details.python-mperceiver.scripts.vision.image_classifierfit\--model.num_latents=32\--model.num_latent_channels=128\--model.encoder.num_frequency_bands=32\--model.encoder.num_cross_attention_layers=2\--model.encoder.num_self_attention_blocks=3\--model.encoder.num_self_attention_layers_per_block=3\--model.encoder.first_self_attention_block_shared=false\--model.encoder.dropout=0.1\--model.encoder.init_scale=0.1\--model.decoder.num_output_query_channels=128\--model.decoder.dropout=0.1\--model.decoder.init_scale=0.1\--data=MNISTDataModule\--data.batch_size=64\--optimizer=AdamW\--optimizer.lr=1e-3\--lr_scheduler.warmup_steps=500\--trainer.accelerator=gpu\--trainer.devices=1\--trainer.max_epochs=30\--trainer.logger=TensorBoardLogger\--trainer.logger.save_dir=logs\--trainer.logger.name=logsModel constructiondescribes how to implement model-specific command line interfaces
with the Lightning CLI. Training checkpoints are written to thelogs/img_clf/version_0/checkpointsdirectory. Assuming
a checkpoint with filenameepoch=025-val_loss=0.065.ckptexists, it can be converted to aperceiver-io🤗 model withfromperceiver.model.vision.image_classifierimportconvert_mnist_classifier_checkpointconvert_mnist_classifier_checkpoint(save_dir="example/mnist-classifier",ckpt_url="logs/img_clf/version_0/checkpoints/epoch=025-val_loss=0.065.ckpt",)so that it can be used in a 🤗 image classification pipelinefromdatasetsimportload_datasetfromtransformersimportpipelinemnist_dataset=load_dataset("mnist",split="test")[:9]images=mnist_dataset["image"]labels=mnist_dataset["label"]classifier=pipeline("image-classification",model="example/mnist-classifier")predictions=[pred[0]["label"]forpredinclassifier(images)]print(f"Labels:{labels}")print(f"Predictions:{predictions}")Labels:      [7, 2, 1, 0, 4, 1, 4, 9, 5]
Predictions: [7, 2, 1, 0, 4, 1, 4, 9, 5]or loaded directly:importtorchfromtransformersimportAutoModelForImageClassification,AutoImageProcessormodel=AutoModelForImageClassification.from_pretrained("example/mnist-classifier")processor=AutoImageProcessor.from_pretrained("example/mnist-classifier")inputs=processor(images,return_tensors="pt")withtorch.no_grad():# use perceiver-io Hugging Face modeloutput_1=model(**inputs).logitswithtorch.no_grad():# or use perceiver-io backend model directlyoutput_2=model.backend_model(inputs.pixel_values)print(f"Predictions:{output_1.argmax(dim=-1).numpy().tolist()}")print(f"Predictions:{output_2.argmax(dim=-1).numpy().tolist()}")Predictions: [7, 2, 1, 0, 4, 1, 4, 9, 5]
Predictions: [7, 2, 1, 0, 4, 1, 4, 9, 5]Seetraining examplesfor more examples.ArticlesArticles referencing this repository:Training compute-optimal Perceiver AR language modelsA gentle introduction to Rotary Position EmbeddingOther implementationsPerceiverPerceiver IOPerceiver AR