Skeleton-of-Thought: Large Language Models Can Do Parallel Decoding
Authors: name: Xuefei Ning; name: Zinan Lin; name: Zixuan Zhou; name: Huazhong Yang; name: Yu Wang
Published: 2023-07-28 06:31:34

Abstract: This work aims at decreasing the end-to-end generation latency of large
language models (LLMs). One of the major causes of the high generation latency
is the sequential decoding approach adopted by almost all state-of-the-art
LLMs. In this work, motivated by the thinking and writing process of humans, we
propose "Skeleton-of-Thought" (SoT), which guides LLMs to first generate the
skeleton of the answer, and then conducts parallel API calls or batched
decoding to complete the contents of each skeleton point in parallel. Not only
does SoT provide considerable speed-up (up to 2.39x across 11 different LLMs),
but it can also potentially improve the answer quality on several question
categories in terms of diversity and relevance. SoT is an initial attempt at
data-centric optimization for efficiency, and reveal the potential of pushing
LLMs to think more like a human for answer quality.