MMFactory: A Universal Solution Search Engine for Vision-Language Tasks
Authors: name: Wan-Cyuan Fan, name: Tanzila Rahman, name: Leonid Sigal

With advances in foundational and vision-language models, and effective
fine-tuning techniques, a large number of both general and special-purpose
models have been developed for a variety of visual tasks. Despite the
flexibility and accessibility of these models, no single model is able to
handle all tasks and/or applications that may be envisioned by potential users.
Recent approaches, such as visual programming and multimodal LLMs with
integrated tools aim to tackle complex visual tasks, by way of program
synthesis. However, such approaches overlook user constraints (e.g.,
performance / computational needs), produce test-time sample-specific solutions
that are difficult to deploy, and, sometimes, require low-level instructions
that maybe beyond the abilities of a naive user. To address these limitations,
we introduce MMFactory, a universal framework that includes model and metrics
routing components, acting like a solution search engine across various
available models. Based on a task description and few sample input-output pairs
and (optionally) resource and/or performance constraints, MMFactory can suggest
a diverse pool of programmatic solutions by instantiating and combining
visio-lingual tools from its model repository. In addition to synthesizing
these solutions, MMFactory also proposes metrics and benchmarks performance /
resource characteristics, allowing users to pick a solution that meets their
unique design constraints. From the technical perspective, we also introduced a
committee-based solution proposer that leverages multi-agent LLM conversation
to generate executable, diverse, universal, and robust solutions for the user.
Experimental results show that MMFactory outperforms existing methods by
delivering state-of-the-art solutions tailored to user problem specifications.
Project page is available at https://davidhalladay.github.io/mmfactory_demo.