CLS-RL: Image Classification with Rule-Based Reinforcement Learning
Authors: name: Ming Li, name: Shitian Zhao, name: Jike Zhong, name: Yuxiang Lai, name: Kaipeng Zhang

Classification is a core task in machine learning. Recent research has shown
that although Multimodal Large Language Models (MLLMs) are initially poor at
image classification, fine-tuning them with an adequate amount of data can
significantly enhance their performance, making them comparable to SOTA
classification models. However, acquiring large-scale labeled data is
expensive. In this paper, we explore few-shot MLLM classification fine-tuning.
We found that SFT can cause severe overfitting issues and may even degrade
performance over the zero-shot approach. To address this challenge, inspired by
the recent successes in rule-based reinforcement learning, we propose CLS-RL,
which uses verifiable signals as reward to fine-tune MLLMs. We discovered that
CLS-RL outperforms SFT in most datasets and has a much higher average accuracy
on both base-to-new and few-shot learning setting. Moreover, we observed a
free-lunch phenomenon for CLS-RL; when models are fine-tuned on a particular
dataset, their performance on other distinct datasets may also improve over
zero-shot models, even if those datasets differ in distribution and class
names. This suggests that RL-based methods effectively teach models the
fundamentals of classification. Lastly, inspired by recent works in inference
time thinking, we re-examine the `thinking process' during fine-tuning, a
critical aspect of RL-based methods, in the context of visual classification.
We question whether such tasks require extensive thinking process during
fine-tuning, proposing that this may actually detract from performance. Based
on this premise, we introduce the No-Thinking-CLS-RL method, which minimizes
thinking processes during training by setting an equality accuracy reward. Our
findings indicate that, with much less fine-tuning time, No-Thinking-CLS-RL
method achieves superior in-domain performance and generalization capabilities
than CLS-RL.