SlowGuess/NeurIPS_2023_OCR huggingface dataset:
22The logos can be website logos, commercial logos used by businesses or companies to represent their brand or product, or logos for organizations or events. In all cases, the label is assigned if the logo is the primary focus of the image.
23These experiments were conducted using a preliminary v1 of the mmc4-core corpus, see this pull request for discussion of small bugfixes in the current v1.1 version.
24Future work would be well-suited to investigate the impact of various flattening schemes on downstream performance; the method described here is just one possible method.
25Similar to [2], we find that training on a maximum of five image sequences can be sufficient for OpenFlamingo models to generalize to 32 shots during inference.
(a) 4-shot
Figure 7: Few shot, in-context MSCOCO captioning performance of OpenFlamingo-3B when training on just captions from LAION-2B vs. mixing inMMC4-core sequences. The model trained onMMC4 sequences is able to generalize to MSCOCO-style captions more effectively vs. the model trained just on LAION-2B image/caption pairs. (Zero shot caption-only=15M caption LAION-2B model)
(b) 8-shot
Table 3: Results of manual verification of 200 randomly sampled documents containing 836 images. A majority of images are topically relevant and well sentence-aligned. The rate of watermarks, ads, duplicates, etc. is low.
% of 836 images
Topically-related	87.7%
Sentence-aligned	80.4%
Has face?	28.3%
Has watermark?	1.6%
Logo-related	3.9%
Ads-related	3.2%
Duplicated	0.7%
Validation CIDEr [35] results for COCO image captioning are in Figure 7. For 4/8-shot in-context learning settings, the model trained on mmc4-core shows 20-30 CIDEr point improvements. The performance of OpenFlamingo-3B trained on just 5M captions/2.5M mmc4 sequences also exceeds a zero-shot application of OpenFlamingo-3B trained on much more data (15M LAION-2B captions); this provides additional evidence that the interleaving in-context setup enables adaptation to MSCOCO-style captions. The performance of the captions-only OpenFlamingo-3B model degrades from 4-shot to 8-shot learning presumably because these longer sequences are significantly different from the single image/captions it's seen at training time.
We introduce mmc4, a corpus of 100M+ documents with 571M images interleaved in 43B English tokens from the popular c4 dataset. Initial experimental results show that models trained on image/text sequences from mmc4 can more effectively perform multimodal in-context learning compared to models trained on single image/captions. We expect interleaving will be important not only for few-shot learning, but also for more diverse multimodal language technologies wherein users may seek to converse with agents with and about visual content in new ways. Future work includes:
1. More precise empirical evaluation of in-context abilities: can models really reason across images/texts in a prompt in flexible ways, or are they limited to interleaved and independent supervised examples?
2. Data scaling: is the performance of in-context vision+language learning bottlenecked by the availability of large-scale interleaved corpora? Or is improved single-modal pretraining sufficient to un-bottleneck multimodal models?
3. Instruction tuning: while interleaving of independent supervised image+text examples enables in-context learning, training an instruction-following multimodal model directly for this case is a promising complementary direction.
We thank the OpenFlamingo team, Sangho Lee, and Jiasen Lu for the helpful discussions, and for being early adopters of mmc4. In addition, we thank Jingkang Yang for helpful discussions inspiring mmc4-core. We thank Stability AI for the compute for the OpenFlamingo experiments. This work was supported in part by DARPA MCS program through NIWC Pacific (N66001-19-2-4031), the
NSF AI Institute for Foundations of Machine Learning (IFML, CCF-2019844), Open Philanthropy, Google, and the Allen Institute for AI.
[28] Sefik Ilkin Serengil and Alper Ozpinar. Hyperextended lightface: A facial attribute analysis framework. In 2021 International Conference on Engineering and Emerging Technologies (ICEET), pages 1-4. IEEE, 2021.
[29] David So, Wojciech Manke, Hanxiao Liu, Zihang Dai, Noam Shazeer, and Quoc V Le. Searching for efficient transformers for language modeling. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, 2021.
[30] Krishna Srinivasan, Karthik Raman, Jiecao Chen, Michael Bendersky, and Marc Najork. WIT: Wikipedia-based image text dataset for multimodal multilingual machine learning. In SIGIR, 2021.
[31] Jun Suzuki, Heiga Zen, and Hideto Kazawa. Extracting representative subset from extensive text data for training pre-trained language models. Information Processing & Management, 60(3):103249, 2023.
[32] Bart Thomee, David A Shamma, Gerald Friedland, Benjamin Elizalde, Karl Ni, Douglas Poland, Damian Borth, and Li-Jia Li. YFCC100M: the new data in multimedia research. Communications of the ACM, 59(2):64-73, 2016.
[33] Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam M. Shazeer, Apoorv Kulshreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, Yaguang Li, Hongrae Lee, Huaixiu Zheng, Amin Ghafouri, Marcelo Menegali, Yanping Huang, Maxim Krikun, Dmitry Lepikhin, James Qin, Dehao Chen, Yuanzhong Xu, Zhifeng Chen, Adam Roberts, Maarten Bosma, Yanqi Zhou, Chung-Ching Chang, I. A. Krivokon, Willard James Rusch, Marc Pickett, Kathleen S. Meier-Hellstern, Meredith Ringel Morris, Tulsee Doshi, Renelito Delos Santos, Toju Duke, Johnny Hartz Søraker, Ben Zevenbergen, Vinodkumar Prabhakaran, Mark Diaz, Ben Hutchinson, Kristen Olson, Alejandra Molina, Erin Hoffman-John, Josh Lee, Lora Aroyo, Ravindran Rajakumar, Alena Butryna, Matthew Lamm, V. O. Kuzmina, Joseph Fenton, Aaron Cohen, Rachel Bernstein, Ray Kurzweil, Blaise Aguera-Arcas, Claire Cui, Marian Croak, Ed Huai hsin Chi, and Quoc Le. Lamda: Language models for dialog applications. ArXiv, abs/2201.08239, 2022.
[34] Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008.
[35] Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. CIDEr: Consensus-based image description evaluation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4566-4575, 2015.
[36] An Yan, Zhankui He, Jiacheng Li, Tianyang Zhang, and Julian McAuley. Personalized show-cases: Generating multi-modal explanations for recommendations. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 2251-2255, 2023.
[37] Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. Opt: Open pre-trained transformer language models. ArXiv, abs/2205.01068, 2022.
A Dataset Card
Our dataset card is available at https://github.com/allenai/mmc4/blob/main/DATASET_CARD.md
B Full Set of LDA Topics
Table 4 contains the full set of topics for the  $k = 30$  LDA model introduced in § 4.
Table 4: LDA[6] topic modeling outputs (k=30 topics) when trained on a random sample of documents from mmc4. Topic frequencies are determined by taking the mean distribution over documents in the corpus. Topic names are generated by GPT-4 conditioned on the top 20 words for each topic, prompted by a request for a short 1-2 word summary.
Topic name	Rate	Top Words
E-commerce	4.61%	products, quality, price, product, online, offer, buy, customers, services, order
Healthcare	2.55%	health, care, body, patients, treatment, medical, pain, cancer, blood, mental
Travel	3.98%	city, hotel, park, visit, travel, trip, tour, enjoy, beach, town
Celebrations	3.94%	fun, wedding, beautiful, christmas, happy, card, birthday, gift, blog, perfect
Music	2.50%	music, band, album, song, sound, songs, dance, show, live, musical
Religion	2.05%	god, church, jesus, lord, faith, man, father, heart, christ, gods
Fashion	4.86%	black, white, size, color, design, wear, style, fabric, cut, fit
Nature	3.05%	water, dog, river, fish, dogs, species, animals, fishing, sea, weather
Geography	3.56%	city, county, state, york, san, north, west, st, john, south
Business	4.15%	management, company, marketing, technology, data, services, team, industry, project, clients
Technology	4.89%	page, app, site, download, website, data, click, google, web, email
Education	2.39%	students, school, learning, skills, children, education, learn, student, training, class
Research	1.43%	data, download, research, analysis, study, al, cells, memory, studies, results
Food	3.31%	food, add, recipe, minutes, chocolate, cream, delicious, chicken, sugar, cheese
Law	2.14%	law, insurance, court, legal, case, state, letter, act, cover, policy
Wellness	1.92%	skin, hair, oil, natural, organic, wine, plant, products, plants, water
Self-improvement	5.27%	change, youre, mind, point, means, fact, thing, ways, question, process
Politics	2.73%	government, president, police, political, war, trump, military, state, party, security
Engineering	2.81%	water, energy, system, power, air, temperature, heat, systems, gas, solar
Sports	3.01%	game, games, team, play, season, players, win, league, player, football
Economy	2.29%	percent, market, million, —, trade, billion, growth, price, company, report
Architecture	3.08%	room, space, house, kitchen, floor, living, pool, building, large, bedroom
Automotive	3.20%	car, vehicle, camera, engine, power, system, model, control, speed, phone
Community	3.91%	community, university, program, research, members, support, development, public, national, group
Finance	1.72%	money, credit, card, real, property, estate, loan, pay, financial, tax
International	2.31%	international, india, countries, china, south, history, united, country, europe, indian
Events	3.93%	2018, event, pm, 2019, 2017, april, 2016, posted, friday, june
Literature	3.73%	book, story, books, film, series, movie, read, characters, stories, reading
Personal	7.96%	ive, didnt, thing, bit, thought, week, wanted, started, pretty, id
Art	2.70%	art, design, de, images, ikea, image, painting, collection, piano, photo
C Most Frequent Top-Level Domains
Table 5 and Table 6 list the top-50 most frequent top-level domains for documents and images as discussed in § 4. We show domain statistics in bothMMC4 andMMC4-core.
Figure 8 shows the top-50 top-level domains of documents in c4- en for reference purposes. The domains are sorted by the frequency of occurrence, as the same with results presented in Figure 6. Previous work also discussed the most represented websites in c4- en ranked by the total number of tokens [12].
Figure 8: The top-50 most frequent top-level domains for documents in c4--en.
D Demonstrative Examples
D.1 Images w/ Watermarks/Ads/Logos
Figure 9a depicts a few sample images containing watermarks in various forms, Figure 9b shows images that are associated with logos, and Figure 9c lists a few sample images related to advertisements. Notice that the dissimilarity between images associated with logos and those pertaining to advertisements is relatively modest. Although images connected to advertisements may occasionally encompass promotional language or persuasive expressions, they may also solely feature logos. Notably, the principal criterion for determining whether an image is ad-related is contingent upon assessing its relevance to the document. If the image is less related to the document, it is more aptly categorized as ad-related. For instance, the interleaved document presented in Table 7 contains two images associated with logos that are intricately linked to the commercial brand being presented within the document. Consequently, these two images are not classified as advertisements.
D.2 Interleaved Document
Table 7 and Table 8 show two interleaved docs from mmc4, displaying the list of sentences and the corresponding assigned images, alongside the CLIP ViT/L-14 image-text similarity score.
Table 5: Top-50 top-level domains for documents in mmc4 and mmc4-core.
mmc4 documents	mmc4-core documents
Domain Name	Percentage	Domain Name	Percentage
www.bbc.com	0.0994%	www.dailymail.co.uk	0.2352%
www.springer.com	0.0993%	www.alibaba.com	0.1601%
www.wikipedia.com	0.0750%	www.indiamart.com	0.1287%
www.nytimes.com	0.0690%	www.teacherspayteachers.com	0.1116%
wwwexpress.co.uk	0.0573%	www.rt.com	0.0858%
www.dailymail.co.uk	0.0530%	www.bbc.com	0.0730%
www. rt.com	0.0519%	www.digital-life.com	0.0728%
itunes.apple.com	0.0508%	www.cbc.ca	0.0673%
www.etsy.com	0.0475%	www.stitcher.com	0.0665%
www.agreatown.com	0.0468%	local.firestonecompleteautcare.com	0.0636%
app-wiringdiagram. herokuapp.com	0.0429%	www.monfrague二线	0.0629%
fineartamerica.com	0.0425%	www.firstpost.com	0.0555%
www.cnn.com	0.0407%	www.express.co.uk	0.0552%
wwwbooking.com	0.0406%	www.androidpolice.com	0.0535%
www Tripadvisor.com	0.0393%	www.usatoday.com	0.0528%
www.firstpost.com	0.0377%	www.auditable.com	0.0481%
www.npr.org	0.0368%	itunes.apple.com	0.0479%
www.wired.com	0.0367%	inhabitat.com	0.0455%
www.breitbart.com	0.0367%	www.cnn.com	0.0435%
www.indiamart.com	0.0364%	www.giftacrossindia.com	0.0433%
www.audible.com	0.0346%	www.houzz.com	0.0428%
medium.com	0.0342%	appadvice.com	0.0421%
www.dailystar.co.uk	0.0338%	www.prweb.com	0.0419%
www.weddingwire.com	0.0336%	wwwtimeout.com	0.0414%
appadvice.com	0.0333%	wccftech.com	0.0412%
www.businessinsider.com	0.0310%	www.ifompt.com	0.0403%
hubpages.com	0.0303%	phys.org	0.0383%
www.Shutterstock.com	0.0285%	www.abc.net.au	0.0381%
www.alibaba.com	0.0282%	www.acahome.org	0.0371%
www.techradar.com	0.0276%	www.npr.org	0.0368%
wwwtimeout.com	0.0265%	www.redmondpie.com	0.0368%
economictimes.indiatimes.com	0.0259%	babyology.com.au	0.0367%
www.prweb.com	0.0256%	www.etsy.com	0.0367%
www.cbc.ca	0.0246%	fgontheweb.com	0.0365%
www.houzz.com	0.0244%	www.pcwworld.com	0.0359%
www.ndtv.com	0.0243%	www.dailystar.co.uk	0.0350%
www.gsmarena.com	0.0243%	www.realtor.com	0.0348%
gizmodo.com	0.0243%	www.wikipedia.com	0.0342%
wn.com	0.0242%	www.advanceduninstaller.com	0.0342%
www.thester.com	0.0240%	shopwizion.com	0.0337%
www/deviantart.com	0.0240%	www.drivermax.com	0.0337%
www.indiebound.org	0.0238%	www.template.net	0.0334%
www.telegraph.co.uk	0.0238%	clemsontigers.com	0.0330%
www.teacherspayteachers.com	0.0236%	www.comparator.in	0.0329%
www.imdb.com	0.0234%	maybeloan.com	0.0320%
sg.carousell.com	0.0233%	medium.com	0.0320%
pixels.com	0.0228%	shoplionly.com	0.0320%
timesofindia.indiatimes.com	0.0227%	wwwreplacement-laptop-battery.com	0.0314%
www.blogtalkradio.com	0.0227%	www.businessinsider.com.au	0.0312%
www.glamour.com	0.0223%	www.dummies.com	0.0312%
Table 6: Top-50 top-level domains for images inMMC4 andMMC4-core. The symbol “*” is employed to denote specific patterns, such as digits or location acronyms, commonly utilized to differentiate sub-sites within the same domain.
mmc4 images	mmc4-core images
Domain Name	Percentage	Domain Name	Percentage
*.bp.blogspot.com	8.7454%	*.bp.blogspot.com	6.8934%
s3.amazon.com	2.1629%	s3.amazon.com	1.8782%
i*.wp.com	1.3176%	images-*.ssl-images-amazon.com	1.7976%
*.staticclickr.com	1.2530%	i*.wp.com	1.4590%
images-*.ssl-images-amazon.com	1.2430%	static*.squarespace.com	0.9530%
static*.squarespace.com	0.8838%	cdn.atwilltech.com	0.9009%
i.pininimg.com	0.6992%	*.staticclickr.com	0.7968%
i.ytimg.com	0.6644%	i.ytimg.com	0.4446%
i*.photobucket.com	0.5075%	*.imimg.com	0.4308%
res.cloudinary.com	0.3683%	bt-photos.global SSL.fastly.net	0.3827%
storage.googleleaps.com	0.3466%	sc*.alicdn.com	0.3700%
i.imgur.com	0.2858%	i.etsystatic.com	0.3494%
lh*.googleusercontent.com	0.2762%	i.pininimg.com	0.3356%
*.bstatic.com	0.2436%	i.dailymail.co.uk	0.2896%
s-media-cache-ak*.pinimg.com	0.2270%	s-media-cache-ak*.pinimg.com	0.2705%
img.youtube.com	0.1954%	i.imgur.com	0.2638%
photos.smugmug.com	0.1934%	i*.photobucket.com	0.2603%
cdn photos sparkplatform.com	0.1915%	lh*.googleusercontent.com	0.2435%
is*-ssl.mzstatic.com	0.1821%	res.cloudinary.com	0.2349%
i.etsystatic.com	0.1727%	is*-ssl.mzstatic.com	0.2142%
odis.homeaway.com	0.1657%	i.bosscdn.com	0.1989%
media-cdn Tripadvisor.com	0.1605%	assets.eflorist.com	0.1927%
media.karousell.com	0.1584%	*.yimg.com	0.1828%
www.picclickimg.com	0.1550%	ecx/images-amazon.com	0.1356%
ae*.alicdn.com	0.1547%	storage.googleleaps.com	0.1329%
m.media-amazon.com	0.1418%	img.youtube.com	0.1192%
ecx/images-amazon.com	0.1385%	cdn.shoplightspeed.com	0.1186%
images.furnituredealer.net	0.1382%	img-aws.ehowcdn.com	0.1163%
image.jimcdn.com	0.1362%	photos.smugmug.com	0.1137%
bt-photos.global.ssl.fastly.net	0.1254%	ecdn.teacherspayteachers.com	0.1047%
t.realgeeks.media	0.1234%	image.jimcdn.com	0.1035%
pbs.twimg.com	0.1194%	m.media-amazon.com	0.1006%
content.cdntwrk.com	0.1126%	cdn.webshopapp.com	0.1000%
www.wikihow.com	0.1106%	i.ebayimg.com	0.0986%
cdn.atwilltech.com	0.1092%	mediad(public broadcasting.net	0.0915%
*.yimg.com	0.1065%	images.template.net	0.0906%
upload.wikimedia.org	0.0960%	ae*.alicdn.com	0.0871%
*.media.tumblr.com	0.0942%	secure.img*-fg.wfcdn.com	0.0861%
f*.bcbits.com	0.0886%	s*.pcdn.co	0.0848%
f.dvipcdn.com	0.0848%	st.hzcdn.com	0.0838%
photos*.blogger.com	0.0833%	assets/simpleviewinc.com	0.0813%
cdn*.weddingwire.com	0.0822%	fgontheweb.com	0.0793%
static/shareasale.com	0.0815%	images.navidirect.org	0.0790%
secure.img*-fg.wfcdn.com	0.0812%	cdni.rtf.com	0.0786%
c*.alamy.com	0.0812%	downloads.intercomcdn.com	0.0777%
usercontent*.hubstatic.com	0.0810%	gallery.mailchimp.com	0.0750%
sc*.alicdn.com	0.0803%	slideplayer.com	0.0690%
static.showit.co	0.0783%	cdn.displays*go.com	0.0677%
i.bosscdn.com	0.0764%	dta*yqvfnsiq.cloudfront.net	0.0660%
*.imimg.com	0.0742%	images.clickdealer.co.uk	0.0644%
(a) Images with watermarks.
(b) Images related to logos.
(c) Images related to ads.
Figure 9: Manually labeled images with watermarks and images related to logos or ads.
Table 7: An example document from mmc4 with interleaved sentences and images, together with the CLIP ViT/-14 image-text similarities. This document contains two logo-related images (the 2nd & 3rd images with "NELO") that are relevant to the content of this document, and are therefore excluded from the category of advertisement.
Sentence	Image	CLIP Similarity
Our new service for teams to manage their fleets for racing.
Getting boats has never been this easy.
Get a step ahead with the planning for your team and get all the boats you need for next season races.	23.51
Our new service for teams to manage their fleets for racing.	22.40
As easy as adding boats to a list, this service aims to be the simplest way to rent boats, no extra knowledge needed and with full support from our staff.
Get all the features of a Nelo boat, from having great equipment to our service team for a fraction of the price of a new boat.	28.76
All our rental boats for racing are carefully maintained and revised between each race so each boat is as good as new.
Table 8: A document instance retrieved from the mmc4 dataset is presented, consisting of interleaved textual sentences and accompanying images, along with the CLIP ViT/-14 image-text similarity scores.
Sentence	Image	CLIP Similarity
Are you thinking about running a retreat for your own group of people?	25.93
We are happy to help you hosting and organizing your own retreat.	19.71
We work with your interest in mind in designing your retreat, and we facilitate the logistics, supporting you all the way for a great experience.	21.29
Nestled within powerful and deeply inspiring nature, in the heart of Tuscany, Italy, Podere Di Maggio is a place born of dreams.	22.35
The dream to be close to and learn from nature.	19.37
The dream to create and share beauty.	19.16
The dream to discover and develop the poetry of being and doing.	18.21
We offer an invitation to explore a wide range of life arts: poetry, dance, music, yoga, meditation, ritual, ceramics, painting, singing, photography, seeing, hearing, touching, feeling, cooking, communicating and collaborating; sharing and daring to discover and unfold yourself.	22.69