Title: CompSRT: Quantization and Pruning for Image Super Resolution Transformers
Primary area: optimization
Keywords: Compression, Quantization, Pruning, Image Super Resolution, Transformer, Swin-IR

Abstract: Model compression has emerged as a way to reduce the cost of using image super resolution models by decreasing storage size and inference time. However, the gap between the best compressed models and the full precision model still remains large and a deeper understanding of compression theory on more performant models remains unexplored. Prior research on quantization of Large Language Models has shown that Hadamard transforms lead to ‘flattened’ weight and activation distributions which lower quantization errors. However, we observe that on SwinIR-light, Hadamard transformations on weights and activations do not lead to flatter distributions, but do lead to lower quantization errors. Instead of flattening distributions, we show that lower errors is caused by the Hadamard transforms ability to reduce the ranges, and increase the proportion of values around 0. Based on these findings, we introduce CompSRT, a more performant way to compress the image super resolution transformer network SwinIR-light. We perform Hadamard-based quantization, and we also perform scalar decomposition to introduce two additional trainable parameters. Our quantization performance statistically significantly surpasses the current state-of-the-art in metrics with gains as large as 1.53 dB, and visibly improves visual quality by reducing blurriness at all bitwidths. At 3-4 bits, to show our method is compatible with pruning for increased compression, we also prune 40\% of weights and show that we can achieve 6.67-15\% reduction in bits per parameter with comparable performance to the state-of-the-art.