first pass over the model card (#1)

- first pass over the model card (d6110ce1f7f896624ad228c39512efed7ec61e0e)


Co-authored-by: Victor Sanh <VictorSanh@users.noreply.huggingface.co>

README.md

@@ -4,20 +4,25 @@ tags:
 - multimodal
 - text
 - image
-license: "apache-2.0"
+license: other
 datasets:
-- obelisc
+- HuggingFaceM4/OBELISC
+- wikipedia
+- facebook/pmd
+- laion/laion2B-en
 ---
 
 
-
+TODO: logo?
 
 # Model Card for m4-80b
 
 <!-- Provide a quick summary of what the model is/does. [Optional] -->
-Some cool model...
-
+ATUM (**A**dapted **T**ransformers for **U**nstructured **M**ultimodal data) is an open-access reproduction of Flamingo, a closed-source visual language model developed by Deepmind. The multimodal model accepts arbitrary sequences of image and text inputs and produces text outputs and is built solely on public available data and models.
+ATUM (TODO) is on par with the original model on various image + text benchmarks, including visual question answering (open-ended and multiple choice), image captioning, and image classification when evaluated with in-context few-shot learning.
 
+The model comes into two variants: a large [80 billion parameters version](https://huggingface.co/HuggingFaceM4/m4-80b) and a [9 billion parameters version](https://huggingface.co/HuggingFaceM4/m4-9b).
+We also fine-tune these base models on a mixture of SFT datasets (TODO: find a more understandable characterization), which boosts the downstream performance while making the models more usable in conversational settings: (TODO: 80B-sfted) and (TODO: 9B sfted).
 
 
 #  Table of Contents
@@ -60,118 +65,92 @@ Some cool model...
 
 # Model Details
 
-## Model Description
-
-<!-- Provide a longer summary of what this model is/does. -->
-Some cool model...
-
-- **Developed by:** HuggingFace
+- **Developed by:** Hugging Face
 - **Model type:** Multi-modal model (text+image)
 - **Language(s) (NLP):** en
-- **License:** apache-2.0
-- **Parent Model:** [laion/CLIP-ViT-H-14-laion2B-s32B-b79K](https://huggingface.co/laion/CLIP-ViT-H-14-laion2B-s32B-b79K) and  [huggingface/llama-65b](https://huggingface.co/huggingface/llama-65b)
-- **Resources for more information:** More information needed
+- **License:** other
+- **Parent Model:** [laion/CLIP-ViT-H-14-laion2B-s32B-b79K](https://huggingface.co/laion/CLIP-ViT-H-14-laion2B-s32B-b79K) and [huggyllama/llama-65b](https://huggingface.co/huggyllama/llama-65b)
+- **Resources for more information:**
     - [GitHub Repo](https://github.com/huggingface/m4/)
-    - Associated Paper: [Flamingo: a Visual Language Model for Few-Shot Learning](https://arxiv.org/abs/2204.14198)
-
-# Uses
-
-<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
-
-## Direct Use
-
-<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
-<!-- If the user enters content, print that. If not, but they enter a task in the list, use that. If neither, say "more info needed." -->
+    - Description of [OBELISC](https://huggingface.co/datasets/HuggingFaceM4/OBELISC): [OBELISC: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documents
+](https://arxiv.org/abs/2306.16527)
+    - Original Paper: [Flamingo: a Visual Language Model for Few-Shot Learning](https://arxiv.org/abs/2204.14198)
 
+ATUM is a large multimodal model that takes sequences of interleaved images and texts as inputs and generates text outputs.
+The model shows strong in-context few-shot learning capabilities (and on par with the closed-source model), and is a robust starting point to fine-tune multimodal models on custom data.
 
+ATUM is built on top of two unimodal open-access pre-trained models to connect the two modalities. Newly initialized parameters in the form of Transformer blocks bridge the gap between the vision encoder and the language model. The model is trained on a mixture of image/text pairs and unstrucutred multimodal web documents.
 
+ 
+# Uses
 
-## Downstream Use [Optional]
-
-<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
-<!-- If the user enters content, print that. If not, but they enter a task in the list, use that. If neither, say "more info needed." -->
-
-
-
-
-## Out-of-Scope Use
-
-<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
-<!-- If the user enters content, print that. If not, but they enter a task in the list, use that. If neither, say "more info needed." -->
-
-
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
 
+The model can be used to perform inference on multimodal (image + text) tasks in which the input is composed of a text query/instruction along with one or multiple images. This model does not support image generation.
 
-# Bias, Risks, and Limitations
+It is possible to fine-tune the base model on custom data for a specific use-case. We note that the instruction-fine-tuned models are significantly better at following instructions and thus should be prefered when using the models out-of-the-box.
 
-<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+The following screenshot is an example of interaction with the model:
 
-Significant research has explored bias and fairness issues with language models (see, e.g., [Sheng et al. (2021)](https://aclanthology.org/2021.acl-long.330.pdf) and [Bender et al. (2021)](https://dl.acm.org/doi/pdf/10.1145/3442188.3445922)). Predictions generated by the model may include disturbing and harmful stereotypes across protected classes; identity characteristics; and sensitive, social, and occupational groups.
+TODO: screenshot
 
 
-## Recommendations
+# How to Get Started with the Model
 
-<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+Use the code below to get started with the model.
 
+<details>
+<summary> Click to expand </summary>
 
+More information needed
 
+</details>
 
 
 # Training Details
 
-## Training Data
+We closel follow the training procedure layed out in [Flamingo](https://arxiv.org/abs/2204.14198). We combine two open-source pre-trained models ([laion/CLIP-ViT-H-14-laion2B-s32B-b79K](https://huggingface.co/laion/CLIP-ViT-H-14-laion2B-s32B-b79K) and [huggyllama/llama-65b](https://huggingface.co/huggyllama/llama-65b)) by initializing new Transformer blocks.
 
-<!-- This should link to a Data Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+The model is trained on the following data mixture of openly accessible data:
 
-More information on training data needed
+| Data Source | Type of Data | Number of Tokens in Source | Number of Images in Source | Epochs | Effective Proportion in Number of Tokens |
+|-------------|----------------------------|------------|----------------------------|--------|
+| PMD | Image-Text Pairs | TODO | TODO | 3 | 73.85% |
+| LAION | Image-Text Pairs | TODO | TODO | 1 | 6.15% |
+| OBELISC | Unstructured Multimodal Web Documents | TODO | TODO | 3 | 2.82% |
+| Wikipedia | Unstructured Multimodal Web Documents | TODO | TODO | 1 | 17.18% |
 
+For multimodal web documents, we feed the model sequences corresponding to the succession of text paragraphs and images.
+For image-text pairs, we form the training sequences by packing images with their captions.
+The images are encoded with the vision encoder and vision hidden states are pooled with Transformer Perceiver blocks and then fused into the text sequence through the cross-attention blocks.
+The training objective is the standard next token prediction.
 
-## Training Procedure
-
-<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
-
-### Preprocessing
-
-More information needed
-
-### Speeds, Sizes, Times
-
-<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
-
-More information needed
 
 # Evaluation
 
 <!-- This section describes the evaluation protocols and provides the results. -->
+We closely follow the evaluation protocol of Flamingo and evaluate ATUM on a suite of downstream image + text benchmarks ranging from visual question answering to image captioning.
+We compare our model to the original Flamingo along with [OpenFlamingo](openflamingo/OpenFlamingo-9B-vitl-mpt7b), another open-source reproduction.
 
-## Testing Data, Factors & Metrics
-
-### Testing Data
+TODO: beautiful plots of shots scaling laws.
 
-<!-- This should link to a Data Card if possible. -->
+TODO: detail of the numbers in a table.
 
-More information needed
-
-
-### Factors
-
-<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
-
-More information needed
 
-### Metrics
+# Bias, Risks, and Limitations
 
-<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
 
-More information needed
+Significant research has explored bias and fairness issues with language models (see, e.g., [Sheng et al. (2021)](https://aclanthology.org/2021.acl-long.330.pdf) and [Bender et al. (2021)](https://dl.acm.org/doi/pdf/10.1145/3442188.3445922)).
+As a derivative of such a language model, ATUM can produce texts that include disturbing and harmful stereotypes across protected classes; identity characteristics; and sensitive, social, and occupational groups.
+Moreover, ATUM can produce factually incorrect texts, and should not be relied on to produce factually accurate information.
 
-## Results
+Here are a few examples of outputs that could be categorized as factually incorrect, biased, or offensive:
+TODO: give 4/5 representative examples
 
-More information needed
+To measure ATUM's ability to recognize socilogical (TODO: find a better adjective) attributes, we evaluate the model on FairFace...
+TODO: include FairFace numbers
 
-# Model Examination
-
-More information needed
 
 # Environmental Impact
 
@@ -184,26 +163,17 @@ Carbon emissions can be estimated using the [Machine Learning Impact calculator]
 - **Cloud Provider:** AWS Sagemaker
 - **Carbon Emitted:** unknown
 
-# Technical Specifications [optional]
+# Technical Specifications
 
-## Model Architecture and Objective
+## Hardware
 
-More information needed
-
-## Compute Infrastructure
-
-More information needed
-
-### Hardware
-
-The training was performed on AWS SageMaker cluster with 64 nodes of 8x80GB A100 GPUs (512 GPUs total). The cluster uses the current EFA network which provides about 340GBps throughput.
+The training was performed on an AWS SageMaker cluster with 64 nodes of 8x80GB A100 GPUs (512 GPUs total). The cluster uses the current EFA network which provides about 340GBps throughput.
 
 As the network is quite slow for the needs of DeepSpeed ZeRO-3 we were only able to clock ~90 TFLOPs.
 
+## Software
 
-### Software
-
-The training software is built on top of HuggingFace Transformers + Accelerate, and DeepSpeed ZeRO-3. Plus [WebDataset](https://github.com/webdataset/webdataset) for data loading.
+The training software is built on top of HuggingFace Transformers + Accelerate, and DeepSpeed ZeRO-3 for training, and [WebDataset](https://github.com/webdataset/webdataset) for data loading.
 
 
 # Citation
@@ -218,15 +188,6 @@ More information needed
 
 More information needed
 
-# Glossary [optional]
-
-<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
-
-More information needed
-
-# More Information [optional]
-
-More information needed
 
 # Model Card Authors [optional]
 
@@ -236,15 +197,4 @@ V, i, c, t, o, r, ,,  , S, t, a, s, ,,  , X, X, X
 
 # Model Card Contact
 
-More information needed
-
-# How to Get Started with the Model
-
-Use the code below to get started with the model.
-
-<details>
-<summary> Click to expand </summary>
-
-More information needed
-
-</details>
+Please open a discussion on the Community tab!