SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Training on different mixture of data Figure 6 shows the result of the first experiment, which consists in training 9B-parameter models on different mixture of data. Training on multimodal web documents allows reaching the same performance using an order of magnitude fewer images than training on image-text pairs, even though the images from the two datasets come from Common Crawl. This underlines the benefit of having longer text contexts for training multimodal models. Moreover, the model trained on multimodal web documents performs better on average. This is particularly striking on visual question-answering benchmarks on which the model trained on image-text pairs slowly degrades through the training. We note, however, that the model trained on image-text pairs has a slight advantage performance-wise in captioning, classification, and OCR tasks (see more details in Appendix A.4.5). We hypothesize that this is due to the nature of image-text pairs: captions can be seen as fuzzy class labels. Last, similarly to Alayrac et al. (2022), we observe that combining the two types of datasets leads to increased performance for a given number of images, tokens, or training compute.
Models trained on OBELICS achieve competitive performance at different scales Following these insights, we show that OBELICS is a viable open alternative to other datasets.
<https://laion.ai/blog/large-openclip/>
Shot	COCO	Flickr30k	VQAv2	OKVQA	TextVQA	VizWiz	VisDial	HatefulMemes
Flamingo-9B	79.4	61.5	51.8	44.7	31.8	22.8	48.0	57.0
OpenFlamingo-9B	0	79.5	59.5	52.7	37.8	24.2	27.5	-	51.6
IDEFICS-9B	46.0	27.3	50.9	38.4	25.9	35.5	48.7	51.8
Flamingo-9B	93.1	72.6	56.3	49.3	33.6	34.9	50.4	62.7
OpenFlamingo-9B	4	89.0	65.8	54.8	40.1	28.2	34.1	-	54.0
IDEFICS-9B	93.0	59.7	55.4	45.4	27.6	36.9	47.9	50.7
Flamingo-9B	99.0	73.4	58.0	50.0	33.6	39.4	51.2	63.9
OpenFlamingo-9B	8	96.3	62.9	54.8	41.1	29.1	38.5	-	54.7
IDEFICS-9B	97.0	61.9	56.4	47.7	27.5	40.4	47.6	51.1
Flamingo-9B	102.2	72.7	59.4	50.8	33.5	43.0	51.3	64.5
OpenFlamingo-9B	16	98.8	62.8	54.3	42.7	27.3	42.5	-	53.9
IDEFICS-9B	99.7	64.5	57.0	48.4	27.9	42.6	-	50.1
Flamingo-9B	106.3	72.8	60.4	51.0	32.6	44.0	50.4	63.5
OpenFlamingo-9B	32	99.5	61.3	53.3	42.4	23.8	44.0	-	53.8
IDEFICS-9B	98.0	64.3	57.9	49.6	28.3	43.7	-	49.8
Flamingo	0	84.3	67.2	56.3	50.6	35.0	31.6	52.0	46.4
IDEFICS	91.8	53.7	60.0	45.2	30.9	36.0	48.9	60.6
Flamingo	4	103.2	75.1	63.1	57.4	36.5	39.6	55.6	68.6
IDEFICS	110.3	73.7	63.6	52.4	34.4	40.4	48.4	57.8
Flamingo	8	108.8	78.2	65.6	57.5	37.3	44.8	56.4	70.0
IDEFICS	114.3	76.6	64.8	55.1	35.7	46.1	47.9	58.2
Flamingo	16	110.5	78.9	66.8	57.8	37.6	48.4	56.8	70.0
IDEFICS	116.6	80.1	65.4	56.8	36.3	48.3	-	57.8
Flamingo	32	113.8	75.4	67.6	57.8	37.9	49.8	55.6	70.0
IDEFICS	116.6	81.1	65.9	57.8	36.7	50.0	-	52.5
Table 2: Performance of IDEFICS against OpenFlamingo and Flamingo. The evaluations were done with random in-context examples, and in an open-ended setting for VQA tasks. (Task, Metric, Query split): (COCO, CIDEr, test), (Flickr30k, CIDEr, test (Karpathy)), (VQAv2, VQA acc., testdev), (OKVQA, VQA acc., val), (TextVQA, VQA acc., val), (VizWiz, VQA acc., testdev), (VisDial, NDCG, val), (Hateful Memes, ROC-AUC, test seen).
We train IDEFICS, an 80 billion parameters Flamingo-like model on a mixture of image-text pairs from LAION (Schuhmann et al., 2022), openly accessible captioning datasets (Singh et al., 2022), OBELICS and multimodal web documents obtained from Wikipedia using a similar extraction strategy. We also train a smaller version of 9 billion parameters, IDEFICS-9B. We compare these models against OpenFlamingo v2 (Awadalla et al., 2023) and Flamingo of the same sizes and trained on a similar mixture of multimodal web documents and image-text pairs. We report the results in Table 2.
IDEFICS is often on par with Flamingo on various multimodal benchmarks. Out of the 8 evaluation tasks, with 32 in-context examples, it either performs better or obtain the same result as Flamingo on 4 of them. At the 9 billion parameter scale, we are still behind Flamingo-9B. However, it is important to highlight that we outperform OpenFlamingo-9B, which was trained on mmc4, in terms of aggregated performance. We achieved a score of 56.5, compared to their score of 55.8, by selecting the best performance across all numbers of in-context examples for each task. This highlights the advantages of OBELICS as an open alternative to a multimodal web document dataset.
With the goal of supporting open-source large multimodal models, we introduce OBELICS, an open web-scale collection of filtered interleaved multimodal web documents based on Common Crawl snapshots. We document a collection and filtering process that balances the scale and removal of undesirable texts and images while addressing some of the well-documented ethical concerns of large-scale multimodal datasets, notably data consent and pornographic content. To demonstrate the usefulness of models trained on multimodal documents, we train IDEFICS on OBELICS and show that it is a viable alternative to closed datasets. Open datasets of multimodal documents with scale, quality, and diversity of sources can help support the ability to train competitive open models.
The authors were granted access to the HPC resources of the Institut du développement et des ressources en informatique scientifique (IDRIS) du Centre national de la recherche scientifique (CNRS) under the allocation 2022-A0121013450 made by Grand équipement national de calcul intensif (GENCI). The initial development of the dataset was done on Jean-Zay cluster of IDRIS, and we thank the IDRIS team for their responsive support throughout the project, in particular Rémi Lacroix. We thank Guillaume Salou for setting up the virtual machines used to download the images of our dataset, and Sebastian Nagel for his valuable assistance in providing insights on Common Crawl. We thank Yacine Jernite and Daniel van Strien for conducting a bias analysis of the models trained on OBELICS.
Abbas, A., K. Tirumala, D. Simig, S. Ganguli, and A. S. Morcos (2023). Semdedup: Data-efficient learning at web-scale through semantic dedduplication.
Aghajanyan, A., B. Huang, C. Ross, V. Karpukhin, H. Xu, N. Goyal, D. Okhonko, M. Joshi, G. Ghosh, M. Lewis, and L. Zettlemoyer (2022). Cm3: A causal masked multimodal model of the internet. ArXiv abs/2201.07520.
Alayrac, J.-B., J. Donahue, P. Luc, A. Miech, I. Barr, Y. Hasson, K. Lenc, A. Mensch, K. Millican, M. Reynolds, R. Ring, E. Rutherford, S. Cabi, T. Han, Z. Gong, S. Samangooei, M. Monteiro, J. L. Menick, S. Borgeaud, A. Brock, A. Nematzadeh, S. Sharifzadeh, M. a. Binkowski, R. Barreira, O. Vinyals, A. Zisserman, and K. Simonyan (2022). Flamingo: a visual language model for few-shot learning. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (Eds.), Advances in Neural Information Processing Systems, Volume 35, pp. 23716-23736. Curran Associates, Inc.
Antol, S., A. Agrawal, J. Lu, M. Mitchell, D. Batra, C. L. Zitnick, and D. Parikh (2015). VQA: Visual Question Answering. In International Conference on Computer Vision (ICCV).
Awadalla, A., I. Gao, J. Gardner, J. Hessel, Y. Hanafy, W. Zhu, K. Marathe, Y. Bitton, S. Gadre, S. Sagawa, J. Jitsev, S. Kornblith, P. W. Koh, G. Ilharco, M. Wortsman, and L. Schmidt (2023). Openflamingo: An open-source framework for training large autoregressive vision-language models.
Bai, Y., A. Jones, K. Ndousse, A. Askell, A. Chen, N. DasSarma, D. Drain, S. Fort, D. Ganguli, T. Henighan, N. Joseph, S. Kadavath, J. Kernion, T. Conerly, S. El-Showk, N. Elhage, Z. Hatfield-Dodds, D. Hernandez, T. Hume, S. Johnston, S. Kravec, L. Lovitt, N. Nanda, C. Olsson, D. Amodei, T. Brown, J. Clark, S. McCandlish, C. Olah, B. Mann, and J. Kaplan (2022). Training a helpful and harmless assistant with reinforcement learning from human feedback.
Beaumont, R. (2021). img2dataset: Easily turn large sets of image ids to an image dataset. https://github.com/rom1504/img2dataset.
Bender, E. M., T. Gebru, A. McMillan-Major, and S. Shmitchell (2021). On the dangers of stochastic parrots: Can language models be too big? Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency.
Biderman, S. and W. J. Scheirer (2020, 12 Dec). Pitfalls in machine learning research: Reexamining the development cycle. In J. Zosa Forde, F. Ruiz, M. F. Pradier, and A. Schein (Eds.), Proceedings on "I Can't Believe It's Not Better!" at NeurIPS Workshops, Volume 137 of Proceedings of Machine Learning Research, pp. 106-117. PMLR.
Biderman, S., H. Schoelkopf, Q. Anthony, H. Bradley, K. O'Brien, E. Hallahan, M. A. Khan, S. Purohit, U. S. Prashanth, E. Raff, A. Skowron, L. Sutawika, and O. van der Wal (2023). Pythia: A suite for analyzing large language models across training and scaling.
M. Dey, M. R. Munoz, M. Masoud, M. Grandury, M. Šaško, M. Huang, M. Coavoux, M. Singh, M. T.-J. Jiang, M. C. Vu, M. A. Jauhar, M. Ghaleb, N. Subramani, N. Kassner, N. Khamis, O. Nguyen, O. Espejel, O. de Gibert, P. Villegas, P. Henderson, P. Colombo, P. Amuok, Q. Lhoest, R. Harliman, R. Bommasani, R. L. López, R. Ribeiro, S. Osei, S. Pyysalo, S. Nagel, S. Bose, S. H. Muhammad, S. Sharma, S. Longpre, S. Nikpoor, S. Silberberg, S. Pai, S. Zink, T. T. Torrent, T. Schick, T. Thrush, V. Danchev, V. Nikoulina, V. Laippala, V. Lepercq, V. Prabhu, Z. Alyafeai, Z. Talat, A. Raja, B. Heinzerling, C. Si, D. E. Taşar, E. Salesky, S. J. Mielke, W. Y. Lee, A. Sharma, A. Santilli, A. Chaffin, A. Stiegler, D. Datta, E. Szczechla, G. Chhablani, H. Wang, H. Pandey, H. Strobelt, J. A. Fries, J. Rozen, L. Gao, L. Sutawika, M. S. Bari, M. S. Al-shaibani, M. Manica, N. Nayak, R. Teehan, S. Albaniae, S. Shen, S. Ben-David, S. H. Bach, T. Kim, T. Bers, T. Fevry, T. Neeraj, U. Thakker, V. Raunak, X. Tang, Z-X. Yong, Z. Sun, S. Brody, Y. Uri, H. Tojarieh, A. Roberts, H. W. Chung, J. Tae, J. Phang, O. Press, C. Li, D. Narayanan, H. Bourfoune, J. Casper, J. Rasley, M. Ryabinin, M. Mishra, M. Zhongybi, M. Peyrounette, N. Patry, N. Tazi, O.Sanseviero,P.von PlatenP.CornetteP.F.Lavallee,R.Lacroix,S.Rajbhandari,S.Gandhi,S.Smith,S.Requena,S.Patil,T.Dettmers A.Baruwa,A.Singh,A.Cheveleva,A.-L.Ligozat,A.Subramonian,A.Nével,C.Lovering D.Garrette,D.Tunuguntla,E.Reiter,E.Taktasheva,E.VoloshinaE.Bogdanov,G.I Winata,H.Schoelkopf,J.-C.Kalo,J.Novikova,J.Z.Forde,J.Clive,J.Kasai,K.Kawamura,L.Hazan,M.Carpuat,M.Clinciu,N.Kim,N.Cheng,O.Serikov,O.Antverg O.van der Wal,R.Zhang,R.Zhang,S.Gehrmann,S.Mirkin,S.Pais,T.Shavrina T.ScialomT.YunT.LimisiewiczV.Rieser,V.ProtasovV.MikhailovY.Pruksachatkun,Y.Belinkov,Z.Bamberger,Z.Kasner,A.RuedaA.PestanaA.Feizpour A.KhanA.FaranakA.SantosA.HeviaA.Unldreaj,A.Aghagol,A.Abdollahi A.Tammour,A.HajiHosseini,B.Behroozi,B.Ajbade,B.SaxenaC.M.Ferrandis D.Contractor,D.Lansky,D.David,D.Kiela,D.A.NguyenE.TanE.BaylorE.Ozoani F.Mirza,F.Ononiwu,H.Rezanejad,H.Jones,I.BhattacharyaI.SolaimanI.Sedenko I.NejadgholiJ.Passmore,J.Seltzer,J.B.SanzL.Dutra,M.SamagaioM.Elbadri M.Mieskes,M.Gerchick,M.Akinlolu,M.McKenna,M.Qiu,M.Ghauri,M.Burynok N.Abrar,N.RajaniN.Elkott,N.FahmyO.SamuelR.AnR.KromannR.Hao S.AlizadehS.ShubberS.WangS.RoyS.ViguierT.LeT.OyebadeT.LeY.Yang Z.NguyenA.R.KashyapA.PalascianoA.CallahanA.ShuklaA.Miranda-Escalada A.SinghB.BeilharzB.WangC.BritoC.ZhouC.JainC.XuC.FourrierD.L PerinanD.MolanoD.YuE.ManjavacasF.BarthF.FuhrimannG.AltayG.Bayrak G.BurnsH.U.VrabecI.BelloI.DashJ.KangJ.GiorgiJ.GoldeJ.DPOSada K.R.SivaramanL.BulchandaniL.LiuL.ShinzatoM.H.de Bykhovetz,M.Takeuchi M.PamiesM.A.CastilloM.NezhurinaM.SangerM.SamwaldM.CullanM.WeinbergM.D.WolfM.MihaljcicM.LiuMFreidankM.KangN.SeelamN.Dahlberg N.M.BroadN.MuellnerP.FungP.HallerR.ChandrasekharR.EisenbergR.Martin R.CanalliR.SuR.SuCahyawijayaS.GardaS.S.DeshmukhS.MishraS.KiblawiS.OttS.Sang-aroonsiriS.KumarS.SchweterS.BharatiT.LaudT.Gigant T.KainumaW.KusaY.LabrakY.S.BajajY.VenkatramanY.XuY.XuZ.TanZ.XieZ.YeM.BrasY.Belkada,and T.Wolf (2023).Bloom:A 176b-parameter open-access multilingual language model..
Xie, S. M., H. Pham, X. Dong, N. Du, H. Liu, Y. Lu, P. Liang, Q. V. Le, T. Ma, and A. W. Yu (2023). Doremi: Optimizing data mixtures speeds up language model pretraining.
Yang, Z., Z. Gan, J. Wang, X. Hu, Y. Lu, Z. Liu, and L. Wang (2022). An empirical study of gpt-3 for few-shot knowledge-based vqa. In Proceedings of the AAAI Conference on Artificial Intelligence, Volume 36, pp. 3081-3089.
Young, P., A. Lai, M. Hodosh, and J. Hockenmaier (2014). From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. Transactions of the Association for Computational Linguistics 2, 67-78.
Yu, J., Z. Wang, V. Vasudevan, L. Yeung, M. Seyedhosseini, and Y. Wu (2022). Coca: Contrastive captioners are image-text foundation models. Transactions on Machine Learning Research.
Yuan, S., S. Zhao, J. Leng, Z. Xue, H. Zhao, P. Liu, Z. Gong, W. X. Zhao, J. Li, and J. Tang (2022). Wudaomm: A large-scale multi-modal dataset for pre-training models.
Yuksekgonul, M., F. Bianchi, P. Kalluri, D. Jurafsky, and J. Zou (2023). When and why vision-language models behave like bags-of-words, and what to do about it? In International Conference on Learning Representations.
Zhang, B. and R. Sennrich (2019). Root Mean Square Layer Normalization. In Advances in Neural Information Processing Systems 32, Vancouver, Canada.
Zhang, J., Y. Zhao, M. Saleh, and P. J. Liu (2019). Pegasus: Pre-training with extracted gap-sentences for abstractive summarization.
Zhang, R., J. Han, A. Zhou, X. Hu, S. Yan, P. Lu, H. Li, P. Gao, and Y. Qiao (2023). Llama-adapter: Efficient fine-tuning of language models with zero-init attention.
Zhang, S., S. Roller, N. Goyal, M. Artetxe, M. Chen, S. Chen, C. Dewan, M. Diab, X. Li, X. V. Lin, T. Mihaylov, M. Ott, S. Shleifer, K. Shuster, D. Simig, P. S. Koura, A. Sridhar, T. Wang, and L. Zettlemoyer (2022). Opt: Open pre-trained transformer language models.
Zhou, Y., Y. Sheng, N. H. Vo, N. Edmonds, and S. Tata (2021). Simplified dom trees for transferable attribute extraction from the web. ArXiv abs/2101.02415.
Zhu, W., J. Hessel, A. Awadalla, S. Y. Gadre, J. Dodge, A. Fang, Y. Yu, L. Schmidt, W. Y. Wang, and Y. Choi (2023). Multimodal C4: An open, billion-scale corpus of images interleaved with text. arXiv preprint arXiv:2304.06939.
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Section 4.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] We think that the release of such a dataset strikes a constructive trade-off between the risks associated with datasets built on top of crawled web pages (for instance, the presence of images with faces, the potential of PII in texts, offensive, insulting or threatening, etc.) with the future works that a dataset of such scale, quality and thoughtful filtering can enable. We further discuss these points in A.3.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes] We read the ethics review guidelines and tried our best to match the expectations. Our content is extracted from publicly available websites at the time of the web crawl. Given the size of our dataset, it would be prohibitive to get the explicit consent of the authors of these websites. Instead, we respect the choice of content creators by removing opted-out images. Such a strategy cannot be exhaustive and we remain available for content creators to opt-out of the dataset.
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)? [Yes] We will release the code used for the creation of the model and its training, along with the model itself.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Appendix A.4.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)? [N/A]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Appendix A.4
(a) If your work uses existing assets, did you cite the creators? [Yes] We mentioned the libraries we used.
(b) Did you mention the license of the assets? [Yes] We only used open-source libraries.
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [Yes] See the ethics review guidelines part.
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [Yes] The dataset we are releasing is built from publicly accessible websites. As such, there is no content in our dataset that hasn't been publicly visible on the web at some point. Similarly, the dataset might contain texts or images that can be considered offensive, insulting, or threatening, as such data is prevalent on the web. We took measures to remove pornographic content and low-quality texts as much as possible. We did not take additional intentional measures to remove personal information. A manual inspection of 250 random samples reveals that there isn't obvious
personally identifiable information (excluding celebrities and people mentioned in news articles), although it is likely that the dataset contains some.