Does Vector Quantization Fail in Spatio-Temporal Forecasting? Exploring a Differentiable Sparse Soft-Vector Quantization Approach

Authors: Chao Chen, Tian Zhou, Yanjun Zhao, Hui Liu, Liang Sun, Rong Jin
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords: spatio-temporal forecasting, vector quantilization, sparse regression, differentiable, soft

Abstract:
Spatio-temporal forecasting is crucial in various fields and requires a careful balance between identifying subtle patterns and filtering out noise. Vector quantization (VQ) appears well-suited for this purpose, as it quantizes input vectors into a set of codebook vectors or patterns. Although VQ has shown promise in various computer vision tasks, it surprisingly falls short in enhancing the accuracy of spatio-temporal forecasting. We attribute this to two main issues: inaccurate optimization due to non-differentiability and limited representation power in hard VQ. To tackle these challenges, we introduce Differentiable Sparse Soft-Vector Quantization (SVQ), the first VQ method to enhance spatio-temporal forecasting. SVQ balances detail preservation with noise reduction, offering full differentiability and a solid foundation in sparse regression. Our approach employs a two-layer MLP and an extensive codebook to streamline the sparse regression process, significantly cutting computational costs while simplifying training and improving performance. Empirical studies on five spatio-temporal benchmark datasets show SVQ achieves state-of-the-art results, including a 7.9\% improvement on the WeatherBench-S temperature dataset and an average MAE reduction of 9.4\% in video prediction benchmarks (Human3.6M, KTH, and KittiCaltech), along with a 17.3\% enhancement in image quality (LPIPS). Code is publicly available at https://anonymous.4open.science/r/SVQ-Forecasting.

Decision: reject
Reviewer scores: 5, 8, 5, 6

Reviews and discussion:
<thread>
creation date:
1730255514392

thread_type:
official_review

<thread_object>
creator: reviewer_1zsw

<content>
summary:
This paper addresses the limitations of traditional Vector Quantization (VQ) and demonstrates impressive performance in spatio-temporal forecasting. SVQ uses a two-layer MLP to approximate sparse regression, reducing computational complexity while maintaining the flexibility to map each input vector to multiple codebook vectors. This soft quantization captures the complex dynamics of spatio-temporal data, preserving essential information while minimizing noise. The experiments confirm that SVQ is an efficient and expressive quantization mechanism applicable to various forecasting tasks. The visualizations provide valuable insights into the behavior and advantages of SVQ.

strengths:
1. Innovative Approach: The paper effectively combines sparse regression with differentiable quantization, addressing the non-differentiability and limited representational power of traditional VQ. Using MLP to approximate sparse regression allows the model to capture complex patterns efficiently.  
2. Simplicity and Effectiveness: The proposed method is intuitive and easy to implement, with straightforward derivations and motivations. It demonstrates significant improvements across multiple tasks and models.  
3. Comprehensive Experiments: The paper provides detailed evaluations of the proposed quantization mechanism, including ablation studies and supplementary materials that address key questions. The well-designed visualizations offer excellent insights into the behavior and strengths of SVQ.

weaknesses:
1. Visual Layout: Perhaps due to space constraints, the layout of the figures and tables could be more aesthetically pleasing.

questions:
1. The paper emphasizes the advantages of the proposed method in spatio-temporal forecasting, but VQ is also widely used in generative tasks (e.g., VQ-VAE). Could this method be applied to such tasks?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730449183852

thread_type:
official_review

<thread_object>
creator: reviewer_akz2

<content>
summary:
The paper introduces a novel Differentiable Sparse Soft-Vector Quantization (SVQ) method, which integrates sparse regression with differentiability to tackle optimization challenges in vector quantization. This approach aims to enhance representation capacity in spatio-temporal forecasting tasks, marking a significant advancement in the field. While the SVQ method presents an innovative approach to vector quantization, the paper would benefit from clearer connections between theory and practice, updated baselines, deeper integration insights, and improved mathematical clarity to strengthen its contributions to the field.
Soundness:2

strengths:
The proposed Differentiable Sparse Soft-Vector Quantization (SVQ) method represents a novel advancement in vector quantization techniques.

weaknesses:
The method does not adequately demonstrate how the theoretical advantages of sparse regression are translated into tangible improvements in quantization performance. While the authors discuss optimization strategies, they fail to provide a clear connection between these theoretical claims and the practical outcomes. A more comprehensive explanation of how these optimization techniques directly enhance quantization would bolster the credibility of their approach.
	In the experimental section, I noticed that the baseline models employed are relatively outdated. Given the recent advancements in spatio-temporal forecasting, particularly the emergence of various diffusion-based methods that have demonstrated significant improvements in predictive performance, it would be beneficial for the authors to consider incorporating these state-of-the-art models as baselines. This would provide a more comprehensive evaluation of the proposed method's effectiveness and advantages.
	The explanation of the quantization module's implementation lacks depth regarding its integration with the overall spatio-temporal forecasting model. While the authors outline the architecture and components involved, they do not provide sufficient details on how the quantization process interacts with other model elements or influences the final forecasting results. A more thorough exploration of these interactions would enhance the clarity and applicability of their proposed method.
	In Section 4, the mathematical proof lacks clarity in the notation used, which may hinder readers' understanding. For example, what’s the meaning of g' after Eq. (8)? Additionally, the proof does not establish a strong connection to the problem being addressed. I recommend revising this section to improve the clarity of the symbols and to explicitly link the proof to the main objectives of the paper.

questions:
See weakness
</content>
</thread_object>

</thread>
<thread>
creation date:
1730713803986

thread_type:
official_review

<thread_object>
creator: reviewer_pbdr

<content>
summary:
Vector quantization (VQ) is insufficient in improving the accuracy of spatiotemporal prediction. This paper introduces differentiable sparse soft vector quantization (SVQ) that can strike a balance between detail preservation and noise reduction, providing a solid foundation for full differentiability and sparse regression. Empirical studies on five spatiotemporal benchmark datasets show that SVQ achieves the best results, including a 7.9% improvement on the WeatherBench-S temperature dataset, a 9.4% average MAE reduction in video prediction benchmarks (Human3.6M, KTH, and KittiCaltech), and a 17.3% improvement in image quality (LPIPS).

strengths:
1. The manuscript proposes a differentiable sparse soft vector quantization (SVQ) method, which is the first vector quantization method applied to spatiotemporal prediction and shows significant improvement.
2. The SVQ method proposed in the manuscript has achieved leading performance in multiple real-world spatiotemporal prediction tasks, significantly reducing errors on multiple benchmark datasets, such as reducing errors by 7.9% on the WeatherBench dataset.
3. The SVQ proposed in the manuscript can be seamlessly integrated into different types of spatiotemporal prediction models as a plug-in, and has improved performance in various architectures, demonstrating the versatility of the method.

weaknesses:
1.The SVQ method proposed in the manuscript still requires a lot of computing resources, especially in the case of high-dimensional data and large-scale codebooks.
2.The comparison methods cited by the author in Tables 1 and 2 are only up to date in 2022, and lack comparisons of the latest methods in the past two years.

questions:
1.Will the SVQ module added as a plug-in to the model have similar performance improvements for other tasks (such as image generation or natural language processing)? What are the applicable application scenarios?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730716217429

thread_type:
official_review

<thread_object>
creator: reviewer_pa7x

<content>
summary:
This paper identifies the limited performance of traditional vector quantization (VQ) in spatiotemporal forecasting due to non-differentiability and limited representation power. It proposes Differentiable Sparse Soft - Vector Quantization (SVQ), which approximates sparse regression with a two-layer MLP for differentiability and uses Soft-VQ with sparse regression to capture patterns and filter noise. Empirical results on multiple datasets show SVQ achieves state-of-the-art performance, is versatile as a plug-in, and effectively balances detail and noise reduction. Ablation studies and additional analyses confirm its key components’ importance and its robustness.

strengths:
The paper shows originality through the development of SVQ, a novel combination of sparse regression and soft vector quantization for spatio-temporal forecasting with theoretical analysis. Empirical validation is extensive, with experiments on multiple real - world datasets and comparisons to existing methods, achieving state-of-the-art results and validating the method's effectiveness and quality. It has potential applications in various domains and can inspire future research, opening new avenues for exploration and providing insights for model development.

weaknesses:
1. The novelty of the approach is limited, and the performance improvement appears marginal. Furthermore, there is no discussion on the additional computational overhead that these slight performance enhancements (such as those observed in wind speed and humidity in Table 1, and in the KTH dataset in Table 2, as well as in Table 18) will incur. A more detailed analysis of the computational overhead introduced by SVQ, compared to baseline methods, is needed, especially for cases where the performance gains are smaller.

2. The motivation is unclear. It is not surprising to use huge codebook and sparse representation to improve the effect, because huge codebook itself brings a lot of extra parameter redundancy. So why use vector quantization? Because in other fields (e.g., video compression, video generation), VQ is to compress redundant information, not to add redundant information. You could further explain their rationale for using vector quantization in this context, given its typical use for compression in other fields.

3. The theoretical analysis provided seems unrelated to the content of the article. Furthermore, the article fails to discuss the relationship between the information or features extracted after compression using SVQ and the original spatiotemporal data. Consequently, there is a notable lack of corresponding theoretical discussion. A more in-depth exploration of how the features learned through SVQ are related to the original spatiotemporal data would greatly aid in fully elucidating the mechanism underlying SVQ.

4. The ablation study conducted is insufficient. There is no doubt that setting the code size to 10000 will yield better performance compared to 1000. A more detailed discussion of the trade-offs involved (such as efficiency, convergence, etc.) with larger code sizes would be helpful.

5. The introduction of redundant over-complete codebooks and additional computational overhead has resulted in a lack of discussion on computational efficiency, speed, and complexity, among other factors. Empirical measurements of training and inference times, memory usage, and computational complexity, as a function of codebook size, would provide a more comprehensive illustration of the advantages of SVQ.

questions:
1. What is the motivation of using vector quantization into spatiotemporal prediction?
2. What is the significance of theoretical analysis in Chapter 4? Is this theoretical analysis related to video prediction?
3. What are the ''improvement'' in Table 1.2.3. refer to, SimVP?
4. Can the method proposed in the paper be compared with diffusion based models? For example, ExtDM: Distribution Extrapolarization Diffusion Model for Video Prediction, CVPR2024. What are the differences between these two methods, e.g., their application scenarios or efficiency?
5. Why are there different types of comparison results between WeatherBench-S and WeatherBench-M in Tab. 1 (Total Cloud Cover in WeatherBench-S and Wind UV in WeatherBench-M)? Why not compare the same subjects? What are the differences in SVQ performance across different physical quantities and data scales?
</content>
</thread_object>

</thread>
<thread>
creation date:
1733190273886

thread_type:
official_comment

<thread_object>
creator: reviewer_akz2

<content>
title:
read authors' responses

comment:
I have read the authors' responses, which helps me better understand their method. I tend to keep my rating.
</content>
</thread_object>

</thread>
<thread>
creation date:
1733246355217

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Author's Summary on the Rebuttal

comment:
We sincerely thank all reviewers for their valuable feedback that helped improve our paper. We also sincerely thank the AC in advance for the considerable time and effort devoted to further reviewing and facilitating the discussion process.

---
> **Summary of Discussion Process**

+ **Resolved Concerns**:

  + Reviewer **PA7x**: The main concerns were the motivation, theoretical analysis, and computational cost. Although the motivation was not fully confirmed, we believe that the latter two concerns were addressed after discussion.

  + Reviewer **pbDr**: The main concerns were the computational cost and the lack of comparisons with the latest methods from the past two years. The reviewer confirmed that all concerns were effectively resolved after discussion.

  + Reviewer **AkZ2**: The main concerns were the theoretical advantages of SVQ and the lack of comparisons with the latest methods from the past two years. These concerns were effectively resolved after discussion. The reviewer appreciated our responses, which helped them better understand our method.

  + Reviewer **1Zsw**: The main concern was the visual layout of the paper. We provided a revised PDF with an optimized layout, and we believe that the layout issue has been addressed.

+ **Remaining Concerns**:

  + Reviewer **PA7x**: The reviewer appreciated that SVQ appears to be a promising VQ technology that addresses some of the issues in existing VQ technologies. But the remaining concerns are:
      + The unclear motivation for applying VQ to video prediction tasks and why we only experimented with video prediction.

      We have clarified that video prediction is not the only task, as we also experimented with other two tasks—weather forecasting and traffic forecasting. Our primary motivation lies in leveraging VQ as a process to reduce noise while preserving the signal. We view this as a general approach applicable to a variety of spatio-temporal forecasting tasks to enhance prediction performance by improving the signal-to-noise ratio. While video prediction task may often emphasize the quality of video generation, SVQ is designed with broader applicability in mind. It aims to address challenges across diverse spatio-temporal forecasting tasks. These tasks frequently require a balance between denoising and preserving meaningful details, a need that aligns closely with the core motivation behind SVQ. Our experimental results in weather forecasting and traffic forecasting further validate this point.

      + The use of a common and unremarkable approach to enhance performance through increased parameters and computation.  

      We clarified that this may be a misunderstanding, as the enhancements observed in our method cannot be attributed significantly, if at all, to increased parameters or computational overhead. Instead, the improvements are primarily attributed to two key factors: first, the differentiable SVQ design, which minimizes quantization error, and second, the enhanced representation power achieved through sparse regression-based VQ.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper was previously submitted to NeruIPS 2024, with title “A Differentiable Sparse Soft-Vector Quantization (SVQ) for Spatio-Temporal Forecasting”, but got rejected due to some weaknesses. The authors didn’t make material improvements except chancing the title into “Does Vector Quantization Fail in Spatio-Temporal Forecasting? Exploring a Differentiable Sparse Soft-Vector Quantization Approach”. Hence, I would recommend rejecting the paper. 

Some comments for the authors to consider:

1.  The proposed Sparse Soft-Vector Quantization (SVQ) is not specific to spatio-temporal forecasting, but instead a general compression method that can be used for various tasks. The authors consider only spatio-temporal forecasting for experimental evaluation though. Since spatio-temporal forecasting is a comprehensive task that can be affected by many components, the presented results are not convincing to demonstrated the superiority of SVQ over its competitors. On the other hand, while evaluating the paper from the viewpoint of spatio-temporal forecasting, the link between SVQ and the problem space is unclear, as pointed out by Reviewer PA7x and Reviewer AkZ2.

2. It seems that the theorems presented in Section 4 are specific to the traditional SVQ, not the proposed differentiable SVQ. 

3. What do you mean by $R_+^m$? Is it to enforce all components being nonnegative? If so, the deductions in Eq(2) and Eq(3) are wrong.