Title: PIPE: Physics-Informed Position Encoding for Alignment of Satellite Images and Time Series in Typhoon Forecasting
Venue: NeurIPS 2025
Accepted: yes
Keywords: AI4Science

Time series

Postion encoding

Climate
Abstract: Multimodal time series forecasting is foundational in various fields, such as utilizing satellite imagery and numerical data for predicting typhoons in climate science. However, existing multimodal approaches primarily focus on utilizing text data to help time series forecasting, leaving the visual data in existing time series datasets underexplored. Furthermore, it is challenging for models to effectively capture the physical information embedded in visual data, such as satellite imagery's temporal and geospatial context, which extends beyond images themselves. To address this gap, we propose physics-informed positional encoding (PIPE), a lightweight method that embeds physical information into vision language models (VLMs). PIPE introduces two key innovations: (1) a physics-informed positional indexing scheme for mapping physics to positional IDs, and (2) a variant-frequency positional encoding mechanism for encoding frequency information of physical variables and sequential order of tokens within the embedding space. By preserving both the physical information and sequential order information, PIPE significantly improves multimodal alignment and forecasting accuracy. Through the experiments on the most representative and the largest open-sourced satellite image dataset, PIPE achieves state-of-the-art performance in both deep learning forecasting and climate domain methods, demonstrating superiority across benchmarks, including a 12\% improvement in typhoon intensity forecasting over prior works.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThey propose a method to align different modalities which is physics informed, taking into account spatial and temporal information. Additionally, a separate positional encoding that uses variant-frequency is introduced for the physical variables. These should ideally help in typhoon intensity forecasting with the ability to integrate satellite imagery, which was not yet done before. The paper shows experiments that illustrate improvement over existing models, including models currently being used for current weather forecasting models. Ablations also show the contribution of each component of the proposed model.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n- Problem is well-motivated\n- Improvement with the addition of satellite image in forecasting\n- Prediction of typhoon positions seem accurate (Fig 3)\n- Ablation study shows contribution of each component of the proposed model\n- Extensive experiments that analyze the different aspects of PIPE such as showing decreasing errors with decreasing leading times, showing model improves with more introduced data (Appendix, Fig 5)\n\nWeaknesses\n- Design choices need to be further explained — i.e., instead of just saying “x was used…”, the rationale behind the choice could be further details (e.g., using Qwen as the text and vision encoders, freezing vision encoder, not freezing text encoder, using different positional indexing depending on the type of data such as sequential, 3D indexing, physics informed indexing, variant-frequency positional encoding)\n- Implementation details are not very clear, especially on the Qwen baseline and how it was trained, and how exactly it differs from PIPE —- is it just the position encoding that’s different?\n\nQUESTIONS:\n- why use Qwen-2.5-vl for the text and vision encoders?\n- what’s the motivation for freezing the vision encoder and not the text encoder?\n- on temporal position IDs: time has a “wrap around” property where the start of the next year is close to the end of the previous year. How is this taken into account (Eq 7)? Related, why are different components using different types of position IDs—why not just use the same type for all inputs?\n- Line 262-263: what is meant by the original dataset publication? is it the Digital typhoon dataset with the typhoon track annotations from Best Track?\n- Is the Qwen-2.5-VL baseline trained the same way as PIPE but without the multimodal alignment enhancements?\n- The PIPE contribution/improvement on the latitude and longitudes don’t seem big perhaps using UTM would better quantify the improvement (Tables 1,2)? 1 degree of lat/lon are quite large so the differences could be better emphasized in UTM since it’s in meters\n\nLIMITATIONS:\nThe paper mostly talks about results having 12h lead time, is this generally enough to warn relevant agencies and for decision-making?\nThe distribution of the data was also not discussed which could pose limitations on the types of locations that this model can be applicable to.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes PIPE (Physics-Informed Position Encoding), a method that improves multimodal time series forecasting by embedding physical metadata—such as time, latitude, and longitude—into a vision-language model. PIPE consists of two key components: a physics-informed positional indexing scheme and a variant-frequency sinusoidal encoding that reflect real-world spatiotemporal structures. Applied to typhoon forecasting using satellite images and numerical data, PIPE outperforms existing models, achieving up to 12% improvement in intensity prediction.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n\n1. The paper clearly formulates a novel multimodal time series forecasting task that integrates satellite imagery with numerical sequences, addressing a realistic and impactful use case in climate forecasting.\n1.  PIPE is easy to implement and requires no additional learnable parameters, making it broadly applicable to existing vision-language models with minimal overhead.\n\n**Weaknesses:**\n1. While PIPE delivers promising empirical gains in multimodal time series forecasting, its core methodological design offers limited novelty. The physics-informed positional indexing maps known physical variables—such as timestamps, latitude, and longitude—to token position IDs using deterministic rules. This approach is structurally similar to absolute and 3D positional encoding schemes previously used in Transformers and Vision Transformers. Likewise, the variant-frequency sinusoidal encoding extends the standard Transformer positional encoding by introducing customized frequencies for each physical variable (e.g., hourly or geographic periodicity). While this results in a tailored encoding of spatiotemporal information, it remains an intuitive modification of the classic sinusoidal formulation.\n1. In addition, because these encodings are non-learnable and differ substantially from the token patterns observed during pretraining, it is unclear how effectively the underlying vision-language model can interpret them. Pretrained models like Qwen-2.5-VL are unlikely to have encountered structured positional patterns grounded in real-world physics (e.g., hourly timestamps mapped to unique negative IDs), and no mechanisms are introduced to ensure the model learns to associate these encodings with physical meaning. The integration thus relies entirely on the model’s ability to infer these semantics during fine-tuning, which may be unreliable. Without interpretability studies showing the model's reliance on the injected encodings, there is a risk that these handcrafted inputs are underutilized.\n1. The evaluation in this paper is limited to a single domain—typhoon forecasting using the Digital Typhoon dataset—which raises concerns about the generalizability of PIPE. Although the dataset is large and well-annotated, it represents only one type of spatiotemporal phenomenon under relatively consistent geographic and temporal conditions. It remains unclear whether PIPE can adapt to other multimodal forecasting tasks that differ in dynamics, data quality, or modality structure, such as precipitation estimation, air quality prediction, urban heat mapping, or traffic forecasting. These applications may involve different temporal resolutions, irregular sampling, heterogeneous sensor data, or weaker correlations between image and numerical modalities. \n1. The paper lacks a thorough comparison to alternative and widely-used positional encoding strategies, which limits the strength of its empirical claims. While ablation studies demonstrate the contribution of PIPE’s components (e.g., physics-informed indexing and variant-frequency encoding), the evaluation does not include baselines using other strong positional encoding methods, such as learned positional embeddings, relative positional encodings, or Rotary Positional Encoding (RoPE), which has proven effective in both language and vision tasks. These methods are well-established, often adaptable, and have demonstrated strong performance in capturing sequence order or spatial structure, including in long-context or multimodal settings. Moreover, since PIPE is non-learnable and fixed, while others are adaptive or trainable, it would be particularly valuable to compare performance, robustness, and interpretability across these approaches.\n\nQUESTIONS:\nThis reviewer would like to recommend that the authors please:\n\n1. Clarify how PIPE is novel compared to existing encoding methods like RoPE, relative position encodings, and 3D indexing.\n1. Provide evidence (e.g., ablations or probing) that the model meaningfully uses the injected physical encodings.\n1. Include experiments on PIPE’s generalizability to domains beyond typhoon forecasting.\n1. Compare PIPE to standard positional encoding baselines such as learned embeddings and RoPE.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nTo address the challenge of effectively integrating images and structured data in multimodal time series forecasting, this paper proposes a Physics-Informed Position Encoding (PIPE) method. PIPE aims to better extract the underlying time and position information embedded in satellite images and enhance the alignment between visual data and time series. The method is applied to complex meteorological scenarios such as typhoon intensity and trajectory forecasting.\n\nSTRENGTHS AND WEAKNESSES:\n1. Advantages:\n\n(1) This paper proposes the PIPE method as a targeted solution for effectively integrating sequential data and satellite imagery. While it is applied to typhoon prediction in this work, the approach has significant practical relevance.\n\n(2) The proposed physics-informed positional indexing method distinguishes image patches from sequential data tokens using negative indices—a novel and elegant design that effectively addresses index conflicts.\n\n(3) PIPE achieves state-of-the-art performance in both deep learning-based forecasting and climate modeling tasks.\n\n2. Weaknesses :\n\n(1) The main contributions of the paper focus on incorporating physical information through positional indexing and encoding at the model input level. Although the paper claims to be physics-informed, the physical guidance is limited to input encoding and does not extend to more meaningful components such as physical knowledge or physics-guided loss functions or constraints on model behavior.\n\n(2) The title of this paper is for alignment of satellite images and time series, but the experiments are restricted to the domain of typhoon forecasting and only one testing dataset, which cannot fully support the main claim and limits the generalization of the proposed method. \n\n(3) The paper does not investigate how PIPE performs across different categories of typhoon intensity. Since typhoons are typically classified into six levels, conducting experiments to analyze the model’s performance across these different intensity levels would provide deeper insights into its robustness and practical utility.\n\n(4) If the evaluation remains limited to a single typhoon dataset, I suggest explicitly mentioning \"typhoon\" in the title to reflect the task more accurately. Moreover, if the focus is indeed on typhoon prediction, the method should be validated on additional typhoon datasets and compared with more typhoon-specific baselines.\n\nQUESTIONS:\n1. Can PIPE be extended to other domains such as remote sensing in agriculture, urban heat island analysis, or sea ice prediction? Have similar tasks been attempted?\n\n2. Have you conducted experiments to evaluate the performance of PIPE across different typhoon intensity levels? I suppose the performance may vary depending on the severity of the typhoon.\n\n3. Apart from using negative indices, have you considered other mechanisms to separate visual and textual tokens (e.g., using token type embeddings)?\n\nLIMITATIONS:\nIt would be helpful to further elaborate on the applicability of the proposed model—whether it is limited to weather-related tasks or potentially effective across a broader range of multimodal applications.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes PIPE (Physics-Informed Positional Encoding), a multimodal time-series forecasting framework that integrates visual information using a VLM. The key contribution is its physics-informed positional encoding, which embeds real-world metadata (latitude, longitude, and timestamp) into visual token positions, enabling better spatiotemporal alignment across inputs. A variant-frequency sinusoidal function further captures the periodic nature of physical variables. PIPE is evaluated on typhoon forecasting, showing that incorporating physical context improves multimodal prediction.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n1.\tThe paper proposes a lightweight method to embed physical information into VLMs, allowing the model to encode shared global physical knowledges in a simple way. This approach has potential benefits for a wide range of transformer-based models in time-series prediction tasks.\n2.\tThe framework makes effective use of historical physical metadata (e.g., latitude, longitude, and time): once as explicit values in the textual input, and again through physics-informed positional indexing applied to visual tokens. \n3.\tThe application to typhoon forecasting is highly relevant and impactful, with potential real-world benefits for disaster preparedness and evacuation planning.\n4.\tThe paper is clearly structured, and easy to follow.\n\nWeakness\n1.\tThe first claimed contribution may be overstated. While the paper introduces a novel positional encoding scheme, the overall architecture closely follows the standard structure of VLMs for multimodal input processing.\n2.\tThe comparison against baselines is potentially unfair. Most baseline models do not incorporate visual data, except for Qwen-2.5-VL-3B. Since the performance gains over Qwen-2.5-VL-3B are relatively small, the strength of the contribution may be overstated without broader comparison to other vision-enabled baselines.\n3.\tWhile the proposed method is demonstrated on typhoon forecasting, other real-world applications could benefit from embedding physical information. If the results in the typhoon domain cannot be reliably reproduced, it would be valuable for the authors to evaluate the method in additional domains to support its generalizability.\n4.\tMinor typographical issue: in line 141, there should be a space between “encoding” and “PIPE (PIPE).”\n\nQUESTIONS:\nSee the comments above. \n\nAdditionally, why does the Figure 1 show two distinct points with the same timestamp (2018/10/29 00:00) but different latitude, longitude, and intensity values?\n\nLIMITATIONS:\nHave the authors adequately addressed the limitations and potential negative societal impact of their work? If so, simply leave “yes”; if not, please include constructive suggestions for improvement. In general, authors should be rewarded rather than punished for being up front about the limitations of their work and any potential negative societal impact. You are encouraged to think through whether any critical points are missing and provide these as feedback for the authors.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces PIPE (Physics-Informed Position Encoding), a framework for multimodal time series forecasting, with a focus on integrating satellite imagery and numerical data for tasks such as typhoon prediction. Unlike existing models that either ignore visual data or only capture pixel-level semantics, PIPE incorporates physical metadata (timestamps, latitude, longitude) directly into the position encoding of vision-language models (VLMs). \nPIPE is evaluated on Typhoon dataset and achieves state-of-the-art forecasting accuracy, notably improving typhoon intensity and track predictions compared to both AI and domain-specific baselines.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\nIntroducing PIPE which explicitly embeds physical attributes—like timestamps, latitude, and longitude—into both the positional indexing and encoding schemes of a vision-language forecasting model. This is beyond traditional positional encoding (which only preserves token order) by incorporating global, real-world geospatial and temporal information directly into the model’s structure. This aligns satellite imagery with time-series data, enabling the model to understand not just visual patterns but also their physical context, crucial for accurate climate forecasting. Authors have performed rigorous experiments to establish their proposals.\n\nWeaknesses\nIt has been tested only on one domain, experiments could be performed on other applications such as crop monitoring, climate change, etc.\n\nAlthough, the title of the paper is physics informed…, it does not modal physics laws, constraints etc.\n\nWhile PIPE innovatively encodes timestamps and geo-coordinates (latitude & longitude), it does not incorporate other rich physical attributes (like atmospheric pressure fields, wind vectors, or physical constraints from fluid dynamics).\n\nThe authors themselves note future work should integrate broader physical laws and constraints to better model real-world dynamics.\n\nWhile PIPE improves short (6–12 hour) forecasts, I am not sure how does it analyze very long-term forecasting performance (e.g., multiple days, weeks ahead) where errors compound and encoding choices might have more dramatic effects.\n\nAuthors could have included the comparison between standard positional encodings and variant -frequency positional encodings.\n\nFew observations:\n\nThe statement in the abstract: “However, existing multimodal approaches primarily ….., leaving the visual data in existing time series datasets untouched.” The integration of time series and vision (multimodal) data is discussed in the following works: Paper 1, Paper 2. But the wording is overly absolute.\n\nIn Figure 1, the axes and legend require clarification. The units for intensity are not explicitly labelled (e.g., hPa)\nIn Figures 4 and 5, the x-axis, y-axis, and legend are not properly visible. \nAlgorithm 1 and related descriptions are insightful, but formatting inconsistencies (e.g., indentation of bullet points, alignment of variables) make it hard to follow. Please refine for clarity.\n\nQUESTIONS:\nPlease see weakness, if you can address them\n\nLIMITATIONS:\nOne paragraph on limitations of the work should be included", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-04"}