Package: palme
🌴 PALM-E: A Multi-Modal AI ModelThis is the open source implementation of the SOTA multi-modality foundation model "PALM-E: An Embodied Multimodal Language Model" from Google, PALM-E is a single large embodied multimodal model, that can address a variety of embodied reasoning tasks, from a variety of observation modalities, on multiple embodiments, and further, exhibits positive transfer: the model benefits from diverse joint training across internet-scale language, vision, and visual-language domains.PAPER LINK: PaLM-E: An Embodied Multimodal Language ModelNoteThis is just the model architecture, no pretrained weights, no tokenizerTo actually conduct inference you would need to --> setup tokenizer for text and images -> train -> inferenceIf you are doing research into multi modal models and would like to train this model and release it open source join the agora lab by clicking on the banner!AppreciationAll the creators in Agora,Join Agorathe community of AI engineers changing the world with their creations.LucidRains for inspiring me to devote myself to open source AI🚀 Quick StartInstallation 📦pipinstallpalmeUsage 🎨importtorchfrompalme.modelimportPalmE#usageimg=torch.randn(1,3,256,256)caption=torch.randint(0,20000,(1,1024))model=PalmE()output=model(img,caption)print(output.shape)# (1, 1024, 20000)TrainingHere is a summary table of the key training hyperparameters mentioned in the paper:HyperparameterValueBatch size2048Learning rate1.5e-4Warmup steps10,000Gradient accumulation steps4Weight decay0.01Dropout rate0.1Embedding dropout rate0.1Attention dropout rate0.1OptimizerAdamWGradient clipping1.0The key details are:Batch size of 2048Learning rate of 1.5e-4 with 10k warmup stepsAdamW optimizerDropout of 0.1 on embeddings, attention, and full modelWeight decay of 0.01Gradient clipping of 1.0They used a fairly standard transformer hyperparameters configuration. The large batch size and gradient accumulation allows them to train huge models.Set the environment variables:ENTITY_NAME: Your wandb project nameOUTPUT_DIR: Directory to save the weights (e.g.,./weights)MASTER_ADDR: For distributed trainingMASTER_PORTFor master port distributed trainingRANK- Number of nodes servicesWORLD_SIZENumber of gpusConfigure the training:Accelerate ConfigEnable Deepspeed 3Accelerate launch train.pyFor more information, refer to theTraining SOP.Dataset StrategyHere is a summary table of the key datasets mentioned in the paper:DatasetTasksSizeLinkTAMPRobotic manipulation planning, VQA96,000 scenesCustom datasetLanguage TableRobotic manipulation planningCustom datasetLinkMobile ManipulationRobotic navigation and manipulation planning, VQA2912 sequencesBased on SayCan datasetWebLIImage-text retrieval66M image-caption pairsLinkVQAv2Visual question answering1.1M questions on COCO imagesLinkOK-VQAVisual question answering requiring external knowledge14,031 questions on COCO imagesLinkCOCOImage captioning330K images with captionsLinkWikipediaText corpusN/ALinkThe key robotics datasets were collected specifically for this work, while the larger vision-language datasets (WebLI, VQAv2, OK-VQA, COCO) are standard benchmarks in that field. The datasets range from tens of thousands of examples for the robotics domains to tens of millions for the internet-scale vision-language data.Contribute || Be Part of the PALM-E Adventure 🤝Your brilliance is needed! Join us, and together, let's make PALM-E even more awe-inspiring:Get Your Copy: Fork the PALM-E repo.Make It Local: Clone your fork.Prep Your Tools: Install the necessities.Discover & Innovate: Dive into the code.Craft Your Magic: Branch and code away.Show & Tell: Push your changes and craft a pull request.🐞 Fixes, 🎨 enhancements, 📝 docs, or 💡 ideas – all are welcome! Let's shape the future of AI, hand in hand.Citation@article{driess2023palme,
  title={PALM-E: An Embodied Multimodal Language Model},
  author={Driess, Danny and Xia, Fei and Sajjadi, Mehdi S. M. and Lynch, Corey and Chowdhery, Aakanksha and Ichter, Brian and Wahid, Ayzaan and Tompson, Jonathan and Vuong, Quan and Yu, Tianhe and Huang, Wenlong and Chebotar, Yevgen and Sermanet, Pierre and Duckworth, Daniel and Levine, Sergey and Vanhoucke, Vincent and Hausman, Karol and Toussaint, Marc and Greff, Klaus and Zeng, Andy and Mordatch, Igor and Florence, Pete},
  journal={arXiv preprint arXiv:2303.03378},
  year={2023},
  url={https://doi.org/10.48550/arXiv.2303.03378}}RoadmapURGENT: Debug Tokenizer, make sure multi-modal inputs work.Create Dataset StrategyUpload Training DocumentationGet Training running with multi-modal