Package: open-flamingo
🦩 OpenFlamingoPaper| Blog posts:1,2|DemoWelcome to our open source implementation of DeepMind'sFlamingo!In this repository, we provide a PyTorch implementation for training and evaluating OpenFlamingo models.
If you have any questions, please feel free to open an issue. We also welcome contributions!Table of ContentsInstallationApproachModel architectureUsageInitializing an OpenFlamingo modelGenerating textTrainingDatasetEvaluationFuture plansTeamAcknowledgmentsCitingInstallationTo install the package in an existing environment, runpip install open-flamingoor to create a conda environment for running OpenFlamingo, runconda env create -f environment.ymlTo install training or eval dependencies, run one of the first two commands. To install everything, run the third command.pip install open-flamingo[training]
pip install open-flamingo[eval]
pip install open-flamingo[all]There are threerequirements.txtfiles:requirements.txtrequirements-training.txtrequirements-eval.txtDepending on your use case, you can install any of these withpip install -r <requirements-file.txt>. The base file contains only the dependencies needed for running the model.ApproachOpenFlamingo is a multimodal language model that can be used for a variety of tasks. It is trained on a large multimodal dataset (e.g. Multimodal C4) and can be used to generate text conditioned on interleaved images/text. For example, OpenFlamingo can be used to generate a caption for an image, or to generate a question given an image and a text passage. The benefit of this approach is that we are able to rapidly adapt to new tasks using in-context learning.Model architectureOpenFlamingo combines a pretrained vision encoder and a language model using cross attention layers. The model architecture is shown below.Credit:FlamingoUsageInitializing an OpenFlamingo modelWe support pretrained vision encoders from theOpenCLIPpackage, which includes OpenAI's pretrained models.
We also support pretrained language models from thetransformerspackage, such asMPT,RedPajama,LLaMA,OPT,GPT-Neo,GPT-J, andPythiamodels.fromopen_flamingoimportcreate_model_and_transformsmodel,image_processor,tokenizer=create_model_and_transforms(clip_vision_encoder_path="ViT-L-14",clip_vision_encoder_pretrained="openai",lang_encoder_path="anas-awadalla/mpt-1b-redpajama-200b",tokenizer_path="anas-awadalla/mpt-1b-redpajama-200b",cross_attn_every_n_layers=1)Released OpenFlamingo modelsWe have trained the following OpenFlamingo models so far.# paramsLanguage modelVision encoderXattn interval*COCO 4-shot CIDErVQAv2 4-shot AccuracyWeights3Bmosaicml/mpt-1b-redpajama-200bopenai CLIP ViT-L/14177.345.8Link3Bmosaicml/mpt-1b-redpajama-200b-dollyopenai CLIP ViT-L/14182.745.7Link4Btogethercomputer/RedPajama-INCITE-Base-3B-v1openai CLIP ViT-L/14281.849.0Link4Btogethercomputer/RedPajama-INCITE-Instruct-3B-v1openai CLIP ViT-L/14285.849.0Link9Bmosaicml/mpt-7bopenai CLIP ViT-L/14489.054.8Link* Xattn interval refers to the--cross_attn_every_n_layersargument.Note: as part of our v2 release, we have deprecated a previous LLaMA-based checkpoint. However, you can continue to use our older checkpoint using the new codebase.Downloading pretrained weightsTo instantiate an OpenFlamingo model with one of our released weights, initialize the model as above and use the following code.# grab model checkpoint from huggingface hubfromhuggingface_hubimporthf_hub_downloadimporttorchcheckpoint_path=hf_hub_download("openflamingo/OpenFlamingo-3B-vitl-mpt1b","checkpoint.pt")model.load_state_dict(torch.load(checkpoint_path),strict=False)Generating textBelow is an example of generating text conditioned on interleaved images/text. In particular, let's try few-shot image captioning.fromPILimportImageimportrequestsimporttorch"""Step 1: Load images"""demo_image_one=Image.open(requests.get("http://images.cocodataset.org/val2017/000000039769.jpg",stream=True).raw)demo_image_two=Image.open(requests.get("http://images.cocodataset.org/test-stuff2017/000000028137.jpg",stream=True).raw)query_image=Image.open(requests.get("http://images.cocodataset.org/test-stuff2017/000000028352.jpg",stream=True).raw)"""Step 2: Preprocessing imagesDetails: For OpenFlamingo, we expect the image to be a torch tensor of shapebatch_size x num_media x num_frames x channels x height x width.In this case batch_size = 1, num_media = 3, num_frames = 1,channels = 3, height = 224, width = 224."""vision_x=[image_processor(demo_image_one).unsqueeze(0),image_processor(demo_image_two).unsqueeze(0),image_processor(query_image).unsqueeze(0)]vision_x=torch.cat(vision_x,dim=0)vision_x=vision_x.unsqueeze(1).unsqueeze(0)"""Step 3: Preprocessing textDetails: In the text we expect an <image> special token to indicate where an image is.We also expect an 
 special token to indicate the end of the textportion associated with an image."""tokenizer.padding_side="left"# For generation padding tokens should be on the leftlang_x=tokenizer(["<image>An image of two cats.
<image>An image of a bathroom sink.
<image>An image of"],return_tensors="pt",)"""Step 4: Generate text"""generated_text=model.generate(vision_x=vision_x,lang_x=lang_x["input_ids"],attention_mask=lang_x["attention_mask"],max_new_tokens=20,num_beams=3,)print("Generated text: ",tokenizer.decode(generated_text[0]))TrainingWe provide training scripts inopen_flamingo/train. We provide an example Slurm script inopen_flamingo/scripts/run_train.py, as well as the following example command:torchrun --nnodes=1 --nproc_per_node=4 open_flamingo/train/train.py \
  --lm_path anas-awadalla/mpt-1b-redpajama-200b \
  --tokenizer_path anas-awadalla/mpt-1b-redpajama-200b \
  --cross_attn_every_n_layers 1 \
  --dataset_resampled \
  --batch_size_mmc4 32 \
  --batch_size_laion 64 \
  --train_num_samples_mmc4 125000\
  --train_num_samples_laion 250000 \
  --loss_multiplier_laion 0.2 \
  --workers=4 \
  --run_name OpenFlamingo-3B-vitl-mpt1b \
  --num_epochs 480 \
  --warmup_steps  1875 \
  --mmc4_textsim_threshold 0.24 \
  --laion_shards "/path/to/shards/shard-{0000..0999}.tar" \
  --mmc4_shards "/path/to/shards/shard-{0000..0999}.tar" \
  --report_to_wandbNote: The MPT-1Bbaseandinstructmodeling code does not accept thelabelskwarg or compute cross-entropy loss directly withinforward(), as expected by our codebase. We suggest using a modified version of the MPT-1B models foundhereandhere.For more details, see ourtraining README.EvaluationAn example evaluation script is atopen_flamingo/scripts/run_eval.sh. Please see ourevaluation READMEfor more details.To run evaluations on OKVQA you will need to run the following command:import nltk
nltk.download('wordnet')Future plansAdd support for video inputTeamOpenFlamingo is developed by:Anas Awadalla*,Irena Gao*,Joshua Gardner,Jack Hessel,Yusuf Hanafy,Wanrong Zhu,Kalyani Marathe,Yonatan Bitton,Samir Gadre,Shiori Sagawa,Jenia Jitsev,Simon Kornblith,Pang Wei Koh,Gabriel Ilharco,Mitchell Wortsman,Ludwig Schmidt.The team is primarily from the University of Washington, Stanford, AI2, UCSB, and Google.AcknowledgmentsThis code is based on Lucidrains'flamingo implementationand David Hansmair'sflamingo-mini repo. Thank you for making your code public! We also thank theOpenCLIPteam as we use their data loading code and take inspiration from their library design.We would also like to thankJean-Baptiste AlayracandAntoine Miechfor their advice,Rohan Taori,Nicholas Schiefer,Deep Ganguli,Thomas Liao,Tatsunori Hashimoto, andNicholas Carlinifor their help with assessing the safety risks of our release, and toStability AIfor providing us with compute resources to train these models.CitingIf you found this repository useful, please consider citing:@software{anas_awadalla_2023_7733589,
  author = {Awadalla, Anas and Gao, Irena and Gardner, Joshua and Hessel, Jack and Hanafy, Yusuf and Zhu, Wanrong and Marathe, Kalyani and Bitton, Yonatan and Gadre, Samir and Jitsev, Jenia and Kornblith, Simon and Koh, Pang Wei and Ilharco, Gabriel and Wortsman, Mitchell and Schmidt, Ludwig},
  title = {OpenFlamingo},
  month        = mar,
  year         = 2023,
  publisher    = {Zenodo},
  version      = {v0.1.1},
  doi          = {10.5281/zenodo.7733589},
  url          = {https://doi.org/10.5281/zenodo.7733589}
}@article{Alayrac2022FlamingoAV,
  title={Flamingo: a Visual Language Model for Few-Shot Learning},
  author={Jean-Baptiste Alayrac and Jeff Donahue and Pauline Luc and Antoine Miech and Iain Barr and Yana Hasson and Karel Lenc and Arthur Mensch and Katie Millican and Malcolm Reynolds and Roman Ring and Eliza Rutherford and Serkan Cabi and Tengda Han and Zhitao Gong and Sina Samangooei and Marianne Monteiro and Jacob Menick and Sebastian Borgeaud and Andy Brock and Aida Nematzadeh and Sahand Sharifzadeh and Mikolaj Binkowski and Ricardo Barreira and Oriol Vinyals and Andrew Zisserman and Karen Simonyan},
  journal={ArXiv},
  year={2022},
  volume={abs/2204.14198}
}