Lumos : Empowering Multimodal LLMs with Scene Text Recognition
Authors: name: Ashish Shenoy; name: Yichao Lu; name: Srihari Jayakumar; name: Debojeet Chatterjee; name: Mohsen Moslehpour; name: Pierce Chuang; name: Abhay Harpale; name: Vikas Bhardwaj; name: Di Xu; name: Shicong Zhao; name: Longfang Zhao; name: Ankit Ramchandani; name: Xin Luna Dong; name: Anuj Kumar
Published: 2024-02-12 19:27:26

Abstract: We introduce Lumos, the first end-to-end multimodal question-answering system
with text understanding capabilities. At the core of Lumos is a Scene Text
Recognition (STR) component that extracts text from first person point-of-view
images, the output of which is used to augment input to a Multimodal Large
Language Model (MM-LLM). While building Lumos, we encountered numerous
challenges related to STR quality, overall latency, and model inference. In
this paper, we delve into those challenges, and discuss the system
architecture, design choices, and modeling techniques employed to overcome
these obstacles. We also provide a comprehensive evaluation for each component,
showcasing high quality and efficiency.