TroL: Traversal of Layers for Large Language and Vision Models
Authors: name: Byung-Kwan Lee; name: Sangyun Chung; name: Chae Won Kim; name: Beomchan Park; name: Yong Man Ro
Published: 2024-06-18 03:42:00

Abstract: Large language and vision models (LLVMs) have been driven by the
generalization power of large language models (LLMs) and the advent of visual
instruction tuning. Along with scaling them up directly, these models enable
LLVMs to showcase powerful vision language (VL) performances by covering
diverse tasks via natural language instructions. However, existing open-source
LLVMs that perform comparably to closed-source LLVMs such as GPT-4V are often
considered too large (e.g., 26B, 34B, and 110B parameters), having a larger
number of layers. These large models demand costly, high-end resources for both
training and inference. To address this issue, we present a new efficient LLVM
family with 1.8B, 3.8B, and 7B LLM model sizes, Traversal of Layers (TroL),
which enables the reuse of layers in a token-wise manner. This layer traversing
technique simulates the effect of looking back and retracing the answering
stream while increasing the number of forward propagation layers without
physically adding more layers. We demonstrate that TroL employs a simple layer
traversing approach yet efficiently outperforms the open-source LLVMs with
larger model sizes and rivals the performances of the closed-source LLVMs with
substantial sizes.