PIPE: Physics-Informed Position Encoding for Alignment of Satellite Images and Time Series in Typhoon Forecasting
Authors: Haobo Li
Eunseo Jung
Zixin CHEN
Zhaowei Wang
Yueya WANG
Huamin Qu
Alexis Kai Hon Lau
Venue: NeurIPS 2025 poster ()
Keywords: AI4Science
Time series
Postion encoding
Climate
Primary area: machine_learning_for_sciences
TL;DR: A paper that incorporates satellite images to augment time series forecasting by introducing physics-informed position encoding.

Abstract: Multimodal time series forecasting is foundational in various fields, such as utilizing satellite imagery and numerical data for predicting typhoons in climate science. However, existing multimodal approaches primarily focus on utilizing text data to help time series forecasting, leaving the visual data in existing time series datasets underexplored. Furthermore, it is challenging for models to effectively capture the physical information embedded in visual data, such as satellite imagery's temporal and geospatial context, which extends beyond images themselves. To address this gap, we propose physics-informed positional encoding (PIPE), a lightweight method that embeds physical information into vision language models (VLMs). PIPE introduces two key innovations: (1) a physics-informed positional indexing scheme for mapping physics to positional IDs, and (2) a variant-frequency positional encoding mechanism for encoding frequency information of physical variables and sequential order of tokens within the embedding space. By preserving both the physical information and sequential order information, PIPE significantly improves multimodal alignment and forecasting accuracy. Through the experiments on the most representative and the largest open-sourced satellite image dataset, PIPE achieves state-of-the-art performance in both deep learning forecasting and climate domain methods, demonstrating superiority across benchmarks, including a 12\% improvement in typhoon intensity forecasting over prior works.

PIPE: Physics-Informed Position Encoding for Alignment of Satellite Images and Time Series in Typhoon Forecasting
Haobo Li Eunseo Jung' Zixin Chen' Zhaowei Wang'
Yueya Wang" Huamin Qu' Alexis Kai Hon Lau"
Department Computer Science Engineering Division of Environment & Sustainability Hong Kong University of Science and Technology h1 hliem@connect. ust. hk
Abstract
Multimodal time series forecasting is foundational in various fields, such as uti- lizing satellite imagery and numerical data for predicting typhoons in climate sci- ence. However, existing multimodal approaches primarily focus on utilizing text data to help time series forecasting, leaving the visual data in existing time se- ries datasets underexplored. Furthermore, is challenging for models to effec- tively capture the physical information embedded in visual data, such satellite imagery's 's temporal and geospatial context, which extends beyond images them- selves. To address this gap, we propose physics-informed informed positional encoding (PIPE), a lightweight method that embeds physical information into vision lan- guage models (VLMs). PIPE introduces two key innovations:
(1) a physics- informed positional indexing scheme for mapping physics to positional IDs, and (2) variant- frriunn-frr positional encoding mechanism for encoding frequeney information of physical variables and sequential order of tokens within the em- bedding space. By preserving both the physical information and sequential order information, PIPE significantly improves multimodal alignment and forecasting accuracy. Through the experiments the most representative and the largest open-sourced satellite image dataset, PIPE achieves state-of oftthe-rrt performance in both deep learning forecasting and climate domain methods, demonstrating su- periority across benchmarks, including a 12% improvement in typhoon intensity forecasting over prior works.
Introduction
Time series forecasting plays crucial role in climate modeling (58). This task involves modeling temporal dependencies to predict future values of a target variable, challenge exacerbated by noise, non-stationarity. and the frequent need to integrate heterogeneous auxiliary data. While traditional methods like Autoregressive Integrated Moving Average (ARIMA) rely on statistical priors [12, deep learning architectures (e.g., LSTMs (13), Transformers 47) have recently dominated the field by learning latent temporal patterns from data. However, these methods still struggle to deliver precise forecasts amid the complexity and scale of real-world data, leaving high-stakes tasks such as typhoon-track prediction continue to have a long way to go.
The rise of large language models (LLMs) type of sequence modeling has introduced new op- portunities for time series forecasting. Although LLMs were originally built for NLP tasks such as text generation (36] and summarization [7], their core objective naturally aligns with time-series
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

N
Tean 1E g0
Ground truth
20
(rtesiyy Lity
Predicted Track
Tiee 30A won
18
Latiidee 1236
linghate 14
Tire M0-81120 9016
16
N
Fowsayy 40 ani/a
Lattake1100 11.0
Longtadi: 1220
S0 Lastake
Logitaee
12"N
E20º
E30ºE
140
150ºE
Figure 1: The multimodal time series forecasting task and the forecasting results for Typhoon Yutu by our PIPE-3B. The leading time is 12 hours and the time gap between neighbouring dots is 12 hours. In multimodal time series forecasting, satellite images can improve the forecasting aceuracy.
forecasting: predicting the next token in a sentence mirrors forecasting the next value in a sequence, both conditioned on historical context. Consequently, existing work adapts LLMs forecasting through tokenization techniques or patching technology to splice time-series segments into model context [62]. More recent work broadens the paradigm by injecting auxiliary instructions or de- scriptions through zero-/few-shot inference [4], in-context learning [32], and text-augmented fore- casting [7]. Several studies push the scope further by incorporating explicit temporal cues, for example, TimeLLM [17] and Tni [29] involve temporal information in prefix- prompts, while AutoTimes embeds timestamps as positional encodings to integrate the temporal information 32
However, existing methods for multimodal time series forecasting, which integrate visual and nu- merical data, face numerous limitations. Integrating visual context, such as satellite imagery, into forecasting is indispensable in climate (48) and other domains [15 52), yet state-of-the-arr vi- sion-language models (VLMs) like GPT-4o (36. Gemini 46) and DW 2 are tuned primarily for general domain multimodal data. Furthermore, their projection layers, the vision encoder from CLIP [41). coos-attention (27]. Q-Former [24], and MLP (28), solely focus on pixel-level semantics and overlook the rich physical metadata (e.g., timestamps and geo-coordinates) embedded in real- world imagery. This omission limits their capacity to improve high-stakes multimodal forecasting tasks. For example, typhoon track prediction with satellite imagery (Figure 1} requires correlating pixel values with the speeifie geophysical attributes (e.g.. latitude, longitude) embedded each pixel. As result, addressing these overlooked physical dimensions beyond the pixel-level values multimodal time-series forecasting not only fills a critical gap in existing alignment methods that only focus the pixel-level values but also introduces new task for multimodal alignment.
To address these challenges, we propose physics-informed positional encoding (PIPE). lightweight method to embed latent physical metadata (c.g.. timestamps, geospatial coordinates) into positional encodings. Unlike traditional positional encodings, which focus solely on the sequence order of tokens within one input instance [4 PIPE encodes shared global physical knowledge (e.g., latitude-longitude relationships consistent across instances) while preserving sequence order information. Specifically, PIPE introduces two key innovations: (1) physics-informed informed positional indexing scheme that maps physics to positional IDs, and (2) variant-frequency positional encod- ing that integrates the attributes of physical variables in the input embedding space. They maintain the original token topology while enabling explicit modeling cies. Experiments on the most representative and the largest open-sourced satellite image dataset for typhoons, Digital Typhoon (22). demonstrate improved cross cross-modas modal integration and forecasting accuracy. PIPE achieves late-of-the-art performance compared to general AI and domain models across multiple benchmarks on the multimodal time series forecasting task.
geographic-temporal dependen-
Our contributions are threefold:
We propose the multimodal time-series forecasting scheme to integrate visual information, where time series data is accompanied by corresponding vision data, extending beyond conventional univariate/muliivriate time series forecasting.

---

method to embed physical knowledge into VLMs. Our method contains two key innovations: (1) -nnfomes positional indexing and (2) variant- frequency positional encoding.
We propose the PIPE
Through comprehensive experiments on the most representative task and the largest open- sourced satellite image dataset, we show an obvious gain (12% for intensity forecasting) after appropriately integrating vision and physics. Through the ablation study, we quantify the benefits of (1) integrating visual data for multimodal time series forecasting (8% for intensity forecasting) and (2) integrating physics knowledge (6% for intensity forecasting).
2 Related Work
2.1 Transformers for Time-series Forecasting
Transformers widely used for time series forecasting, demonstrating superior performance traditional statistical models and RNN 42 architectures. Key innovations driving this success in- clude efficient attention mechanisms and architectural adaptations tailored temporal patterns. Re- cent works have introduced several enhancements to address computational complexity and domain- specific challenges. Informer [T tackles the quadratic complexity of standard self-attention through ProbSparse attention combined with distillation operations to prioritize crucial temporal features. Autoformer (57) integrates decomposition from time-series analysis with autocorrelation: inspired attention and outperforms self-attention in both efficiency and accuracy. iTransformer [31] applies the attention and network on the inverted dimensions for time series forecasting. One Fits All 62 fine-tunes on all major types of tasks involving time series. Other variations on transformer 53 TimeMixer [50] etc.
include CrossFormer [51), TimeXer
The patching paradigm has inspired multiple variants. PatchTST [34] segments time series into local windows as input tokens while maintaining channel independence for multivariate data. Building on this concept, works such as Pathformer [5] and Sageformer (59) research transformer-basee patching technology in terms of multiscale and inter-series dependencies. Notably, works such One Fits All [62] and Time-LLM, [17] demonstrate the transferability of patching strategies by adapting pre- trained large language models to time-series forecasting through input token alignment.
However, these developments underscore the challenge of managing complexity when incorporating additional modules, such patch-based components. Our method incorporates physical information via Position IDs, avoiding the need for extra models.
2.2 Multimodal LLMs for Time Series Forecasting
Recent advances in LLMs have catalyzed efforts to develop multimodal models capable of process- ing diverse data modalities (e.g., text, images, audio) through unified architectures. This paradigm has inspired time-series forecasting adaptations that integrate textual instructions with temporal data. TimeLLM [17] reprograms the input time series with text timestamps as prefix-prompts to align the two modalities. Unitime (29 utilizes prefix-prompts
encode frequency information of tempo- ral data to augment the model. AutoTimes (32) uses the embedding of textual timestamps as the position encoding incorporate temporal information. Subsequent works like UrbanGPT [25) TEST 45). ChatTime [49] and GPR4MTS [66] utilize similar methods, aligning text instructions and time series for the augmentation of time series forecasting.
However, for time series forecasting, existing multimodal approaches focus narrowly on aligning textual instructions with numerical time series, neglecting critical vision modalities inherent many forecasting scenarios, such as typhoon forecasting, Our work researches the utilization vision data for time series forecasting.
2.3 Position Encoding in Transformers
Transformers require explicit position encoding to capture sequential order information, unlike RNNs that inherently model temporal relationships through hidden state propagation. Current posi- tion encoding strategies can be categorized into two primary paradigms:
hhats//githubbcco//oooooe//rr/
3

---

1. Absolute position encodes the absolute position of a unit within sentence. The original Trans- former architecture 477 introduced two variants: 1) Learned positional embeddings during training stages. 2) Fixed sinusoidal functions:
pos
PE (pioo sin
1000022/dmoade
pos
P (i(pos,21+1) = cos(
1000026/4made
where denotes the dimension, pos is the position, and dodddl the dimension of embeddings. This matrix simply added to the embeddings before they are fed to the Transformer model. Subsequent methods have been proposed to address the challenges of long sequences efficiency [39
120 30 and improve
2. Relative position encodes the position of a unit relative to other units. Shaw al. 43 pioneered this approach by modifying self-attention to compute relative position biases. Transformer XL [8] introduces recurrence-ane untied position embeddings to add relative position embeddings through additive scalar biases. Wu al. [54] propose to incorporate the real distances between tokens re-scale the raw self-attention weights. Rotary Position Embedding (RoPE) 44 injects relative positions via rotation matrices.
position encoding for long-context modeling. Ke et al. [19] propose
Though effective for local sequence modeling, these methods focus intra-instance positional re- lationships within individual input samples. For time-series forecasting tasks where cross-instance physical dependencies are critical (e.g.. all instances share the global knowledge of geographic in- formation), existing approaches fail capture global temporal-spatial correlations across the entire dataset. Our work addresses this limitation through physies-in -nforreed position encoding. By coding global timestamps with geographic coordinates (Latitudellongitude), our method preserves continuous spatiotemporal relationships across independent time-series sequences.
3 Method
This section formalizes the multimodal time series forecasting problem and proposes physics- informed positional encoding that integrates physical information into VLMs for multimodal time series forecasting. A schematic overview of the method is provided in Figure 2
3.1 Multimodal Time Series Forecasting Problem Formulation
We address the problem of multimodal time series forecasting, where historical observations com- prise both time series data of multiple variables and visual images. Given a sequence of historical time steps:
++: (x:-H Ht Tt- H4- D1) E "xxC
where H denotes the historical time steps, C' the number of variates, along with a corresponding sequence of H images: 4z- tt E Hmg x Wiig for each time step with Himg: W Wing as the height and width of the image, the objective is to forecast the future F time steps:
F++1++F = ( 2442, D1+F) FxC
Our task is to propose VLM model as a cross-modal forecaster (-) to model cross-modal relationships between the multivariate sequence x, 141 and visual sequence i, 14 Formally, we seek to learn:
44114F = fVL 4-1411)
(4)
3.2 VLMS for Multimodal Time Series Forecasting
To perform the multimodal time series forecasting, we use the VLM to encode the time series input and the vision input, following the practice of VL.''s pipeline. this paper, we use the open-source Qwen-2.5-vl 2 as it incorporates 3D positional encoding, which enhances its multimodal abilities.

---

Output series: 980.1, 980.5, 990.3, 9911
Outputs:
LLM
Inputs:
+
Task Instruction
Image
Input series: 980.1, 980.5, 990.3
Position IDs ffor 2D RoPE):
Inc
① Physics- informedd indasiigg
Variant Frequency
Vision Encoder djs
10300 Howee
(lat (laf Fruge)
Klat Ireat
(lat2
10000
iron
:
(lat
Kaat Indh
Riight
In.g
f
Laput
Positional Enoodling
Emibediing
Figure 2: The framework of physics-informed positional encoding. It includes: (1) physics- informed positional indexing scheme that maps physics to positional IDs, and (2) variant- variant-frequency positional encoding that integrates the attributes of physical variables in the input embedding space.
Text embedding To leverage the capability of the pretrained LLM, tokenize the time series data, 2t- 2t-H+1ih H-t into tokens and concatenate them with task-specific instructions (e.g., "Predict next 24 hours of typhoon track"). They fed into the LLM's transformer layers, as depicted with purple inputs in Figure 2] The LLM is trained during the training stage. The prompt design for multimodal time series forecasting can be found in Appendix A
Vision embedding Each image E XaigXangWWon is split into N non-overlapping patches {Pt, N 1==1 where Pt, E 3x3228x22 28, These patches are encoded using the pretrained vision encoder of Qwen-2.5-vl, producing embeddings that dimensionally consistent with the text tokens, as depicted with blue inputs in Figure 2 The vision encoder is frozen during the training stage.
3.3 PIPE
We propose PIPE incorporate physical information into multimodal alignment for multimodal time series forecasting. Our proposed PIPE includes two cores: physics-informer positional index- ing (Figure 2 ()) and variant-frequency positional encoding FFigure 2).. The algorithm can be found in Appendix B
3.3.1 Physics-Informne Positional Indexing
We propose chysics-informen positional indexing to integrate physical information into the model.
Schemes of indexing position IDs. Incorporating physical information into the model using po- sitional IDs provides direct solution without the need for additional structural complexity. We explore three indexing strategies to facilitate this integration:
(1) Sequential indexing. The most intuitive approach is to follow the standard transformer prac- tice [47] and VIT [1] using the sequence index the position IDs. In this scheme, the positional IDs are indexed linearly as:
mositionjids 1, seg_len 1]
(5)
5

---

where seq_len represents the total length of the input sequence, including both text tokens and vision tokens. This approach effectively encodes ID sequential order but lacks explicit order information of the image (2D) video (3D) for multimodal inputs.
(2) 3D indexing. Building Qwen-2.5-VL. 2], this method expands positional indexing to include three independent dimensions: temporal, height, and width, for the alignment of images and videos.
Text tokens continue to use sequential indexing described in Equation 5| while vision tokens are indexed based on their temporal and spatial attributes.
Temporal positions of vision tokens are calculated as:
t t= tokens Bectenss_ee____e per__eer____ee__ per_second x temporal patch_pptchhppe fps
(6)
where tokens_ Biceens_pr__ieeee per_perr_ppooe soc_scoond dictates how many time steps are conceptually packed into second interval of the video, temporal pptpcoorr_ptcc_siieeeee is the number of frames, and fps is the video's frame rate.
For spatial dimensions of vision tokens, the height and width positional IDs correspond to a patch grid ranging from (0, 0) to (Nrow 1, Neot - 1), where Nrow and Noot are the numbers of image patching in height and width, respectively. Although this 3D indexing scheme aligns temporal and spatial order within vision tokens, it only captures the intra-relationship of positions with the input instance. It does not explicitly encode extra-physical properties such as time, latitude, and longitude, which are global knowledge among all instances the dataset.
(3) Physics-Infone Informed positional indexing (Figure 2).
To address the limitations of 3D indexing, we propose a novel physics-informed positional indexing scheme that explicitly integrates global knowledge of physical attributes into positional IDs.
Text embeddings continue to use the sequence indexing scheme described in Equation 5
Temporal positional IDs vision tokens are computed based on the hourly progression of a given year. Specifically, the temporal position ID) is calculated by:
== ddy x 24 + thour
where Eday is the day of the year (ranging from 0 to 365) and thour represents the hour of the day (ranging from 0 23). This indexing introduces meaningful temporal patterns aligned with real-world time progression.
The height and width positional IDs of vision tokens are determined using the latitude and longi- tude of the image patch centers.
To prevent the performance decreases caused by the conflicts between the physical information of vision tokens and the order information of text tokens (refer to the ablation experiment /eecionn 4.6). we map the range of vision positional IDs (t: 8784 (8784 hours year), lat: 180, Ing: 0 - 360) to negative values. This avoids overlap with the text positional ID range, ensuring smooth multimodal integration. Moreover, temporal, latitudinal, and longitudinal dimensions are inherently independent, eliminating concerns about overlap.
After incorporating the cross-instance physical information among all input samples using physics- informed positional indexing, we apply RoPE [44] on position IDs to encode intra-instance posi- tional relationships within individual input samples.
3.3.2 Frriant-Frequency Positional Encoding
We also merge the information of the physical variables into input embeddings. To differentiate between physical variables, we modify the standard sinusoidal positional encoding (Equation I) by introducing a variant-frequenn sinusoidal function.
Pariant-frequent sinusoidal function This function modifies the sine and cosine components and the target dimension based on the temporal, latitude, and longitude frequencies. Figure 2 illus- trates the setting, [quation 9 in the Appendix gives the complete definition, and Figure 4 visualizes

---

the function. For conciseness, the function for image tokens can be formulated as:
pos 2T
PEepos.22 26) sin(
(8)
10000 dmoddl
p
pos
PE PB(pss.2it1) = cos(
10000 2d00ttt
where pos can be Idays lhour, lul, and Ing depending the dimensions. Iday is the day of the year (ranging from to 365) and thour represents the hour the day (ranging from 0 to 23). lat the latitude of the image token, and Ing is the longitude of the image token. p represents the wavelength specific to physics. For temporal data, Pday 366 and Phour = 24 and for spatial dimensions, Platitude. = 180 and Plongitude 360. After the modification, the wavelengths form geometric progression from p to 10000/2 for vision data.
Text tokens preserve the standard sinusoidal encoding to maintain compatibility with pretrained LLM structures. These variant-frequency position encodings are added the input embeddings at the bottom of the decoder stacks after they are divided by dmndde. They map different physical variables to distinet frequency domains before incorporating them into the input embedding space.
Experiments
This section presents a systematic evaluation of the proposed method for the most representative multimodal time series forecasting task, typhoon forecasting. We first describe the datasets, baseline methods, and evaluation metries, followed by the experiments and ablation studies.
4.1 Dataset
For multimodal time series forecasting, utilize the open-source Digital Typhoon dataset (22]. the longest hourly satellite imagery collection dedicated to typhoon analysis spanning 40+ years (1978-2023) with a km spatial resolution. The spatial coverage of the dataset the Western North Pacific basin. The dataset includes 1,116 typhoon sequences and 192,956 images (resolution of 512x512 and resized 224x224). The size of the dataset is different from the size the original paper since the dataset is being regularly updated.
Typhoon track annotations, including intensity, latitude, and longitude. sourced from the Best Track dataset [23]. It the best estimate, a globally recognized benchmark derived from retrospec- tive post-event analysis. This metadata ensures reliable spatiotemporal grounding, it synthesizes all available observational data to reconstruct each typhoon's 's lifecycle with high precision. In our experiments, we will forecast three variables: intensity, latitude, and longitude. The dataset is split using a ratio of 0.7:0.15:0.1 based on the typhoon sequences as the original dataset.
Baselines
Domain models For typhoon forecasting, we compare our method against the state-of-th of-the-art domain-specific NWP-based model: forecasting system of the European Centre for Sredium-Range Weather Forecasts (ECMWF) I0] and two environment-domain large models, Pangu 3 and Gen- Cast [40] which serve as domain-specific benchmarks. Additionally, we include comparisons with the domain practice method, Typhoon Intensity Forecasting based on the SHIPS method (TIFS) [5 We report only the available performance from their paper and do not retrain the models, as we can- not reproduce these domain models.
AI models We train the state-of-the-ae AI models with our dataset, including Transformer- based models (PatchTST (34] iTransformer B1. Crossformer [5T], TimeXer (53) and linear-base mod- els (TiDE 19), LLsed model (One Fits ALL 622, AutoTimes 32) ), and other models (Times- Net (55), TimeMixer (50).. Due to their model design, they do not incorporate visual data. For
the visual data integration, we include benchmark results reported in the original dataset publication
(only the leading time of 12h is available) [22] closed-source Gemini-2.55flash 6 (zero-shot), and
train the original Qwen-2.5-VI. 2
7

---

Table 1: Multimodal time series forecasting results (leading time is 6h).
Intensity (hPa)
Latitude (")
Longitude ())
Distance (km)
Models
MAE RMSE
MAE RMSE
MAE
RMSE
MAE
ECMWF- HRES 10
27.181
domain
PanGu
3
32.892
GenCast
40
20.331
TIFS 35
7.292
PatchTST 34
1.806
2.867
0.199
0.266
0.322
0.404
44.537
iTransformer
1.848
2.979
0.164
0.231
0.203
0.281
31.248
Crossformer
2.389
3.599
0.310
0.418
0.520
0.684
71.216
wo vision
TimeXer (53
3.037
4.523
0,306
0.411
0.411
0.538
59.720
TiDE 9
1.724
2.819
0,161
0.224
0.237
0.312
34.068
One Fits All
62
1.849
2.976
0.170
0.239
0.211
0.290
32.450
AutoTimes 32
1.911
3,088
0.190
0.265
0.279
0.364
40,036
TimesNet 55
2.401
3.711
0.465
0.630
0.855
1.124
113.718
TimeMixer
1,913
2.973
0.177
0.237
0.238
0.313
35.374
vision
Gemini-2.55(lash 6
1.924
3.654
0.174
0.254
0.237
0.798
36,006
Qwen-2.5-VL-3B
1.617
3.231
0.087
0.162
0.103
0.187
17.129
PIPE-3B
1.515
2.981
0.084
0.159
0.095
0.178
16.275
Implementation Details
Both our method and baselines use the same temporal settings with the their
same length of input and output sequences (12h). For One Fits All and AutoTimes, official implementations. Other models without vision, their implementations are through the pub- licly available Time-SSerie--Librrry (56). For the Qwen-2.5-VL. model and PIPE, we LLama- Factory [60] for their implementation. More implementation specifics, including hyperparameters and training protocols, detailed in Appenddix C
4.3 Evaluation Metrics
In NLP tasks, metrics such as ROUGE (26] and BLEU 37 are commonly employed the metrics. In our cases, we focus on numerical output. Specifically, for forecasting intensity, latitude, and Root Mean Square Error (RMSE) and Mean Absolute Error (MAE) primary metrics. When the model accurately predicts these numerical values based on satellite images, we consider it to have effectively aligned the satellite imagery with the time series data. Additionally, we use geographiclib [18] to calculate the position error typhoon tracks based on the latitude and longitude, following the domain practice.
longitude,
4.4 Main Results
Short-term forecasting plays crucial role in enabling timely decision-making. Therefore, the fore- casting performance multimodal time series models is presented for lead times of hours (Table T} and 12 hours (Table 5). The best results are emphasized in bold, while the second-best results are marked with underline.
Overall, our method achieves state-of-the-an performance across the majority of evaluation metrics, demonstrating the efficacy the proposed PIPE integrating physical information during mul- timodal alignment. For the 6-hour lead time Tabll 1), our model outperforms baselines in most metries. For example, it shows 12% improvement of MAE for typhoon intensity forecasting when compared to the best w/o vision models TiDE. The sole exception is the RMSE for intensity fore- casting, where TiDE and PatchTST exhibit marginally superior performance. These results show effectiveness of our approach. A critical observation is the consistent superiority of models incorpo- rating vision data over unimodal alternatives. This finding emphasizes importance of leveraging multimodal inputs to enhance forecasting accuracy in complex spatiotemporal tasks.

---

Paeditted Calondd
Packiced Coaet Golondd
Fird Paar Cokrnd)
1321
Intensity
Latitude
Longitude
333
Figure 3: The visualization for the regression results between predicted values and true values (lead- ing time is hours). (a) Plots for intensity. (b) Plots for latitude. (c) Plots for longitude.
4.5 Regression Analysis
The regression results of all test typhoon sequences (Figure 3) demonstrate that our method achieves accurate typhoon predictions. Notably, the model exhibits superior performance in location forecast- ing compared to intensity forecasting, which may be attributed to the richer spatial information pro- vided by satellite imagery for tracking movement, Additionally, the model shows better predictive performance when typhoon intensity weaker (i.e., higher central pressure, around 1000 hpa).
4.6 Ablation Study
The results of the ablation study are presented in Table 2 and Table 6 with the best performance highlighted in bold. We systematically evaluate three critical components: vision inclusion, physics- informed position indexing, and variant-frequeecc sinusoidal function.
The gain of aligning vision The inclusion of satellite vision data yields significant improvements in forecasting accuracy. Specifically, the MAE for intensity forecasting improves by up 8% when the leading time is set to hours. This demonstrates that cross-modal learning effectively leverages spatial patterns in satellite imagery complement time series data.
Comparison of schemes of indexing position IDs Our physics-informed indexing scheme ad- dresses the critical challenge of preserving physical knowledge while avoiding token order conflicts. To assess its effectiveness, we compare different schemes for indexing position IDs. Specifically, evaluate the performance by (a) removing the 3D indexing scheme (replacing it with sequential indexing), and (b) removing physics-informed indexing while retaining the 3D indexing scheme. The results show that while sequential indexing and 3D indexing perform similarly, both exhibit noticeable performance degradation (6% for MAE of intensity forecasting) compared to the physics- informed indexing scheme. Avoiding the overlap between the physical information of vision tokens and the order information of text tokens is critical. There is a dramatic performance decrease when they share overlapping ranges (e.g., longitude: D-360 and text tokens: 0 D-seqten (seqien is the num- ber of text tokens)). By mapping the position IDs vision tokens to negative values, we preserve the physical information and resolve such conflicts, leading to improved performance.
The gain of integrating physical variables' frequency The incorporation of frequency charac- teristics of physical variables improves physical variable modeling. We show the importance by removing the entire sinusoidal function and only removing the fariant-frequency sinusoidal func- tion. The results reveal that our designed sinusoidal function plays a crucial role in aligning the model with the frequency information of physical variables. Its inclusion enhances the model's ability to leverage these variables effectively. leading to improved performance.
The necessity of combining physics-informed indexing and variant frequency positional em- beddings In terms of track error, PIPE achieves an error of 16.275, compared to 16.860 when using only physics-inforned indexing, 17.177 when using only variant frequency positional embed- dings, and 17. 129 for the baseline. These results clearly demonstrate that the combination both components delivers the most significant performance improvement.

---

Table 2: The results of the ablation study (leading time is 6h).
Intensity (hPa)
Latitude (*)
Longitude (*)
Distance (km)
Models
MAE
RMSE
MAE
RMSE
MAE
RMSE
MAE
w/o vision
1.646
3.220
0.088
0.160
0.102
0,193
17.235
w/o 3D indexing (using sequence)
1.628
3.749
0.087
0.163
0.102
0.185
17.084
w/o physics-informed indexing (using 3D)
1.617
3.231
0.087
0.162
0.103
0.187
17.129
w/o negative indexing
1.961
3.926
0.206
0.360
0.388
0,674
53.548
w/o entire sinusoidal function
1.545
3.053
0.085
0.157
0.097
0,180
16.554
w/o varian--frquency sinusoidal function
1.639
3.178
0.086
0.161
0.101
0.183
16,860
w/o entire physicx-informed indexing
1.581
2.994
0.087
0.161
0.102
0.186
17.117
PIPE-3B
1.515
2.981
0.084
0.159
0.095
0.178
16.275
Every component contributes the multimodal time series forecasting, with vision aligmment pro- viding complementary visual patterns, the physics-informed indexing scheme ensuring physical knowledge integration, and the variant-frequency sinusoidal function incorporating physical vari- ables' frequency information.
4.7 Generalizability Experiment
To demonstrate the generalizability of our method, we add an additional experiment in the Aus- tralian region (AU) 221. The settings remain the same as the main experiment. The results are shown in Table 3 The results demonstrate that our method can be applied to various regions for typhoon forecasting. Even the model trained on the West Pacific region can perform well the new dataset (zero-shot). For instance, compared to Qwen-2.5-V - (after tuning), PIPE-3B (Zero-shot) achieved a lower intensity RMSE (2.773 vs. 2.806) and a smaller distance error (19.090 vs. 19.718). Additionally, PIPE-3B (after tuning) outperformed both models, achieving the best performance.
Table
The results of the AU Typhoon Experiment (leading time is 6h).
Intensity (hPa) MAE
Latitude (*)
Longitude (9)
Distance (km)
Models
RMSE
MAE
RMSE
MAE
RMSE
MAE
1wwn-22.5-V VB (after tuning)
1.399
2.806
0.098
0.201
0.121
0.211
19.718
PIPE-3B (Zero-shot)
1.361
2.773
0.094
0.171
0.118
0.206
19.090
PIPE-3B (after tuning)
1.352
2.558
0.093
0.163
0.116
0.202
18.835
Limitation
While the integration of satellite imagery improves forecasting accuracy, increases the compu- tational complexity needed to process high-resolution images. To address these limitations, future work will focus on improving the efficiency of integrating vision data into forecasting models to enable longer input sequences and extended forecasting horizons for VLMs. Furthermore, we will explore the incorporation of physical laws or constraints. Beyond embedding physical informa- tion, integrating domain-specific physical principles or environmental constraints could improve the model's s interpretability and robustness.
6 Conclusion
This paper proposes a multimodal time series forecasting task and addresses the challenge brought by integrating satellite imagery. Existing approaches only focus on pixel-level features, overlook- ing the rich temporal and geophysical context embedded within vision data. We propose physics- informed position encoding (PIPE). Experimental results demonstrate that PIPE achieves state- of-the-art performance across multiple benchmarks. Ablation studies further validate the distinct contributions of each component. Future work will explore the integration additional physical domain knowledge, such physical laws and constraints, to enhance real-world applicability.
10

---

Acknowledgements
This project is partially supported by RGC TRS grant T22-607/24N and grant from the Re- search Grants Council of the Hong Kong Special Administrative Region, China (Project No. RMGS20RG01).
References
[1] Samira Abnar and Willem Zuidema. Quantifying attention flow in transformers. In Dan Ju- rafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault, editors, Proceedings of the 58th An- nual Meeting of the Association for Computational Linguistics, pages 4190-4197, Online, July 2020.
Association for Computational Linguistics.
[2] Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang. Jun Tang. Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li. Jian- qiang Wan, Pengfei Wang. Wei Ding. Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical 2025.
report. arXiv preprint arXiv:2502.
[3] Kaifeng Bi. Lingxi Xie, Hengheng Zhang, Xin Chen, Xiaotao Gu, and Qi Tian. Accurate medium-range global weather forecasting with 3d neural networks. Nature, 619(7970):533- 538, 2023.
[4] Ching Chang, Wei- Yao Wang, Wen-Chih Peng, and Tien-Fu Chen. Llm4ts: Aligning pre- trained ilms as data-efficient time-series forecasters. ACM Trans. Intell. Syst. Technol., 16(3), April 2025.
[5] Peng Chen, Yingying Zhang, Yunyao Cheng, Yang Shu, Yihang Wang, Qingsong Wen, Bin Yang, and Chenjuan Guo. Pathformer: Multi-scale transformers with adaptive pathways for time series forecasting. arXiv preprint arXiv:2402.05956, 2024.
[6] Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv preprint arXiv:2507.0626. 2025.
[7] John Dagdelen, Alexander Dunn, Sanghoon Lee, Nicholas Walker, Andrew S Rosen, Gerbrand Ceder, Kristin Persson, and Anubhav Jain. Structured information extraction from scientific text with large language models. Nature Communicationss 15(1):1418, 2024.
[8] Zihang Dai, Zhilin Yang. Yiming Yang. Jaime Carbonell, Quoc Le, and Ruslan Salakhutdinov. TrL: Attentive language models beyond fixed-length context. In Anna Korho- nen, David Traum, and Lluis Mårquez, editors, Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2978-2988, Florence, Italy, July 2019. Association for Computational Linguistics.
[9] Abhimanyu Das, Weihao Kong, Andrew Leach, Shaan Mathur, Rajat Sen, and Rose Yu. Long- term forecasting with tide: Time-series dense encoder. arXiv preprint arX1v:2304.08442 084424, 2023.
[10] IFS DOCUMENTATION
Part Ensemble prediction system. 2020.
[11] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, al. An image is worth 16x 16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020.
[12] Siu Lau Ho and Min Xie. The use of arima models for reliability forecasting and analysis. Computers & industrial engineering, 35(1-2):11-2166 1998.
[13] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 1835:1735-7780, 1997.

---

[14] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang. Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3, 2022.
[15] Neal Jean, Marshall Burke, Michael Xie, W Matthew Alampay Davis, David B Lobell, and Stefano Ermon. Combining satellite imagery and machine learning to predict poverty, Science, 353(6301):790-799 2016.
[16] Furong Jia, Kevin Wang, Yixiang Zheng, Defu Cao, and Yan Liu. Gpt4mts: Prompt-based large language model for multimodal time-series forecasting. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 23345,3335, 2024.
[17] Ming Jin, Shiyu Wang, Lintao Ma, Zhixuan Chu, James Zhang, Xiaoming Shi, Pin- Pin-Yu Chen, Yuxuan Liang, Yuan-Fang Li, Shirui Pan, and Qingsong Wen. Time-LLM: Time series fore- casting by reprogramming large language models. In International Conference on Learning Representations (ICLR), 2024.
[18] Charles FF Karney. Algorithms for geodesics. Journal of Geodesy, 87:43-55, 2013.
[19] Guolin Ke, Di He, and Tie- Yan Liu. Rethinking positional encoding in language pre-training.
In International Conference on Learning Representations, 2021.
[20] Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. In International Conference on Learning Representations, 2020.
[21] Asanobu Kitamoto, Erwan Dzik, and Gaspar Faure. Machine learning for the digital typhoon dataset:
Extensions to multiple basins and new developments in representations and tasks. arXiv preprint arXiv:2411. 16421, 2024.
[22] Asanobu Kitamoto, Jared Hwang, Bastien Vuillod. Lucas Gautier, Yingtao Tian, and Tarin Clanuwat, Digital typhoon: Long-term satellite image dataset for the spatio-temporal modeling of tropical cyclones. Advances in Neural Information Processing Systems, 14440633-40633
2023.
[23] Kenneth R Knapp, Michael C Kruk, David H Levinson, Howard J Diamond, and Charles Neumann. The international best track archive for climate stewardship (ibtracs) unifying trop- ical cyclone data. Bulletin the the American Meteorological Society, 91(3):363-376 2010.
[24] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language- image pre-training with frozen image encoders and large language models. In International conference on machine learning, pages 19730-19742. PMLR, 2023.
[25] Zhonghang Li, Lianghao Xia, Jiabin Tang, Yong Xu, Lei Shi, Long Xia, Dawei Yin, and Chao Huang. Urbangpt: Spatio-temporal large language models. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 5351-5362, 2024.
[26] Chin-Yew Lin. ROUGE: A package for automatic evaluation of summaries. Text Summa- rization Branches Out, pages 81, Barcelona, Spain, July 2004. Association for Computa- tional Linguistics.
[27] Hezheng Lin, Xing Cheng, Xiangyu Wu, and Dong Shen. Cat Cross attention in vision transformer. In 2022 IEEE international conference on multimedia and expo (ICME), pages 1-6. IEEE, 2022.
[28] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Ad- vances in neural information processing systems, 36:34892-34916. 2023.
[29] Xu Liu, Junfeng Hu, Yuan Li, Shizhe Diao, Yuxuan Liang, Bryan Hooi, and Roger Zimmer- mann. Unitime: A -empoweee unified model for cross-domain time series forecast- ing. In Proceedings of the ACM Web Conference 2024, pages 4095-4106, 2024.
[30] Xuanqing Liu, Hsiang-Fu Yu, Inderjit Dhillon, and Cho-Jui Hsieh. Learning to encode po- sition for transformer with continuous dynamical model. In Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 6335- PMLR, 13-18 Jul 2020.
12

---

[31] Yong Liu, Tengge Hu, Haoran Zhang, Haixu Wu, Shiyu Wang, Lintao Ma, and Mingsheng Long. itransformer: Inverted transformers are effective for time series forecasting. arXiv preprini arXiv:2310.0662. 2023.
[32] Yong Liu, Guo Qin, Xiangdong Huang, Jianmin Wang, and Mingsheng Long. Auto- times: Autoregressive time series forecasters via large language models. arXiv preprint arXiv:2402.02330, 2024.
[33] Iya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017.
[34] Yuqi Nie, Nam H Nguyen, Phanwadee Sinthong, and Jayant Kalagnanam. A time series is worth 64 words: Long-term forecasting with transformers. arXiv preprint arX1::211..14730, 2022.
[35] Marika Ono, Shoji Notsuhara, Junya Fukuda, Yohko Igarashi, and Kotaro Bessho. Operational use of the typhoon intensity forecasting scheme based on ships (tifs) and commencement of five-day tropical cyclone intensity forecasts. ENE, 128(40):128-7, 2019.
[36] OpenAl et al. Gpt-4 technical report, 2024.
[37] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: method for automatic evaluation of machine translation. In Proceedings the 40th annual meeting of the Association for Computational Linguistics, pages 31 311-318, 2002.
[38] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf. Edward Yang. Zachary De Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperativ style, high- performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' d'Alché- Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
[39]
Ofir Press, Noah A. Smith, and Mike Lewis. Shortformer: Better language modeling using shorter inputs. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli, editors, Proceed- ings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume I: Long Papers), pages 5493-5505, Online, August 2021. Association for Computational Linguisties.
[40] Ilan Price, Alvaro Sanchez-
Ferran Alet, Tom Andersson, Andrew El-Kadi, Do- minic Masters, Timo Ewalds, Jacklynn Stott, Shakir Mohamed, Peter Battaglia, et al. Proba- bilistic weather forecasting with machine learning. Nature, 637(8044): 90, 2025.
[41] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748 8763. PmLR, 2021.
[42] David E Rumelhart, Geoffrey E Hinton, and Ronald Williams. Learning representations by back-propagating errors. nature, 323(6069):533--36 1986.
[43] Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. Self-attention with relative position repre- sentations. In Marilyn Walker, Heng Ji, and Amanda Stent, editors, Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguis- ties: Human Language Technologies, Volume (Short Papers), pages 464-468, New Orleans, Louisiana, June 2018. Association for Computational Linguistics.
[44] Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding, Neurocomput, 568(C), February 2024.
[45] Chenxi Sun, Hongyan Li, Yaliang Li, and Shenda Hong. TEST: Text prototype aligned em- bedding to activate L.M'' ability for time series. The Twelfth International Conference on Learning Representations, 2024.
13

---

[46] Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptisn Baptiste Alayrac Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312. 1805, 2023.
[47] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[48] Mark Veillette. Siddharth Samsi, and Chris Mattioli. Sevir: A storm event imagery dataset for deep learning applications in radar and satellite metcorology. Advances in Neural Information Processing Systems, 33:22009-22019 2020.
[49] Chengsen Wang, Qi Qi. Jingyu Wang. Haifeng Sun, Zirui Zhuang, Jinming Wu, Lei Zhang, and Jianxin Liao. Chattime: A unified multimodal time series foundation model bridging numerical and textual data. In Proceedings of the AAAI Conference Artificial Intelligence, volume 39, pages 1694-127722 2025.
[50] Shiyu Wang, Haixu Wu, Xiaoming Shi, Tengge Hu, Huakun Luo, Lintao Ma, James Y Zhang, and Jun Zhou. Timemixer: Decomposable multiscale mixing for time series forecasting, arXiv preprint 2rXiv:2405.1400 2024.
[51] Wenxiao Wang, Wei Chen, Qibo Qiu, Long Chen, Boxi Wu, Binbin Lin, Xiaofei He, and Wei Liu. Crossformert+:: A versatile vision transformer hinging on cross-scale attention. IEEE Transactions on Pattern Analysis and Machine Intelligence, 46(5):3123-3136) 2023.
[52] Yihe Wang, Yu Han, Haishuai Wang, and Xiang Zhang. Contrast everything: A hierarchical contrastive framework for medical time-series. Advances in Neural Information Processing 3ystems, 36:5599457779 2023.
[53] Yuxuan Wang. Haixu Wu, Jiaxiang Dong, Guo Qin, Haoran Zhang, Yong Liu, Yunzhong Qiu, Jianmin Wang, and Mingsheng Long. Timexer: Empowering transformers for time series forecasting with exogenous variables. arXiv preprint 2.0. 2024.
[54] Chuhan Wu, Fangzhao Wu, and Yongfeng Huang. DA-transformer: Distance- aware trans- former. In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani- Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou, editors, Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, pages 2068, Online, June 2021. Association for Computational Linguistics.
[55] Haixu Wu, Tengge Hu, Yong Liu, Hang Zhou, Jianmin Wang. and Mingsheng Long. Timesnet: Temporal 2d-variation modeling for general time series analysis. arXiv preprint arXiv:2210.021 186, 2022.
[56] Haixu Wu, Tengge Hu, Yong Liu, Hang Zhou, Jianmin Wang, and Mingsheng Long, Timesnet: Temporal 2d-vaiittion modeling for general time series analysis. In International Conference on Learning Representations, 2023.
[57] Haixu Wo, Jiehui Xu, Jianmin Wang, and Mingsheng Long. Autoformer: Decomposition transformers with auto-correlation for long-term series forecasting. Advances in neural infor- mation processing systems, 34:22419-22433 2021.
[58] Yuan Yuan and Lei Lin. Self-supervised pretraining of transformers for satellite image time se- ries classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing,
2020.
[59] Zhenwei Zhang. Linghang Meng, and Yuantao Gu. Sageformer: Series-aware frame- work for long-term multivariate time-series forecasting. IEEE Internet of Things Journal, 11(10)18435-184488 2024.
[60] Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, Zheyan Luo, Zhanghhi Feng, and Yongqiang Ma. Llamafactory: Unified efficient fine-tuning of 100+ language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations), Bangkok, Thailand, 2024. Association for Computational Linguistics.
14

---

[61] Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. Informer: Beyond efficient transformer for long sequence time-series forecasting. In Proceedings of the AAAI conference on artificial intelligence, volume 35, pages 11 1110--11115, 2021.
[62] Tian Zhou, Peisong Niu, Liang Sun, Rong Jin, et al. One fits all: Power general time series analysis by pretrained Im. Advances in neural information processing systems, 36:43322- 43355, 2023.
15

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We state our claim and contributions in our abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitation in section
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The au- thors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the ap- proach. For example, a facial recognition algorithm may perform poorly when image resolution is low images taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to ad- dress problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
16

---

Justification:
We do not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should clearly stated or referenced in the statement of any theo- rems.
The proofs can either appear in the main paper the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be comple- mented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclu- sions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We disclose the implementation details in section 4 and Appendix C for re- producing the results. We also provide the code and data in supplementary materials for reproduction.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken make their results reproducible verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, other means that are appropriate the research performed.
While NeurIPS does not require releasing code, the conference does require all sub- missions to provide some reasonable avenue for reproducibility, which may depend on the nature the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results way produce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which thors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing verifying the results.
5. Open access to data and code
17

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
We provide the code and data the supplementary materials.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (nttps://nips.cr
for more details.
While we encourage the release code and data, we understand that this might not be possible, so "No" is acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
//appsscc/pbbllc...//cc www.c//gwugueeeee Sttp//CooeSbuiieee bommsssiii.........m......m.mmmm onPolicy)
more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification: Please refer to sossetiion 4 and Appenddi for the training and test details. We introduce settings, training hyperparameters, optimizer, etc.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropri- ate information about the statistical significance of the experiments?
Answer:
[Yes]
Justification:
We report standard deviations with three random seeds in subsection E.2
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi-
dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
18

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should prefer- ably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide the computation cost in Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't t make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
We have checked the NeurIPS Code Ethics and our research conforms with
it.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Ye]]
Justification: We discuss the broader impacts of our work in Appendix F including the impacts on other domains and scenarios.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
19

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact spe- cific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitiga- tion strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our paper poses such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by re- quiring that users adhere to usage guidelines restrictions access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer:
[Yes]
Justification:
We properly cite the assets including dataset and code in section 4 and Ap-
pendix C
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
20

---

If assets are released, the license, copyright information, and terms of use in the pack- age should be provided. For popular datasets, pattastts.paperswiiiae paperswithcooeeeo com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the li- cense of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and the documenta- tion provided alongside the assets?
Answer: [Yes]
Justification:
We provide the readme file for our code in the supplementary materials.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
For crowdsourcing experiments and research with human subjects, does the pa- per include the full text instructions given to participants and sereenshots, applicable, well as details about compensation (if any)?
Question:
Answer: [NA]
Justification: Our work does not involve crowdsourcing nor research with human subjects,
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contri- bution of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, cura- tion, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper deseribe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equiva- lent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
21

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: We do use LLMs for the core method development.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM for what should or should be described.
A
Prompts Design
We show prompt design for the multimodal time series forecasting, taking instance of Ty- phoon Yutu as an example.
System Prompt & Task Instruction
You are a typhoon forecasting expert. Below are the past 12 hours of typhoon data and the corresponding satellite images. Your task is forecast the hourly data of the typhoon for the next 12 hours, providing the forecast latitude, longitude, pressure in the same format as the past data format.
Past Data
The corresponding satellite images are: <image> <image> <image> <image> <image> <im- age> <image> <image> <image> <image> <image> <image>, The historical hourly data
from 2018-10-23 01:00:00 to 2018-10-23 12:00:00 is (latitude: [11.65, 11.7, 11.75, 11.8, 11.85, 11.9, 11.95, 11.99, 12.04, 12.09, 12.14, 12.2], longitude: [151.61, 151.41, 151.2, 150.99, 150.79, 150.6, 150.42, 150.26, 150.11, 149.97, 149.83, 149.7], pressure: [974.2, 973.3, 972.5, 971.7, 970.8, 970.0, 967.5, 965.0, 962.5, 960.0, 957.5, 955.0]].
Label Data
The forecast hourly data is: (latitude: [12.26, 12.34, 12.42, 12.5, 12.6, 12.7, 12.81, 12.93,
13.05, 13.17, 13.29, 13.4), longitude: [149.57, 149.44, 149.31, 149.18, 149.04, 148.9, 148. 76, 148.61, 148.46, 148.31, 148. 15, 148.0), pressure: [954.2, 953.3, 952.5, 251.7,950.8, 950.0, 945.8, 941.7, 937.5, 933.3, 929.2, 925.0J.)
22

---

Image Data
B
Method
B.1 Algorithms
In this section, we present the algorithms for PIPE (algorithm 1j. which integrates physical infor- mation into VLMs. To begin, we extract the required physical information (algorithm 2} from the time series data corresponding to each satellite image. This information includes the timestamp and geocoordinates for the typhoon's 's eye. Additionally, since satellite images are divided into patches, we calculate the geocoordinates for the center of each patch. Next, we incorporate this physical information into the positional encoding of VLMs through physics-informet positional indexing algorithm 3). Beyond indexing, we adapt the sinusoidal function by introducing puriant-frequenge sinusoidal encoding (algorithm 4), which embeds the frequency attributes of the variables into the positional embedding. This enhanced positional embedding is then added to the input embedding of the corresponding image tokens. Finally, these integrations utilized predict the next token, enabling the model to leverage both spatial and physical context effectively.
Algorithm 1: PIPE
Require: Time series input corresponding image Ensure: Next token prediction T next
1: T texts T image tokenizer(æ)), vision encoder(2)
Tokenization
2: t, lat, Ing t get (x,pppyysicx,.a Timage)
Extract physical info algorithhm 2
Lat, lat,Ing E R I Xxxee
3: ids "osition_innexingg" teart T Timage lat, Ing)
Compute physics-informed indices
algorithm 3 ids I23xleen(Teen (erentttt Timage))
4: PE t- vi Jun(os,, i)
> Generate variant-frequency position embedding (algorithm 4
PE E Blen (T Teoat Timage Xdmodet
5: IE + Tiezt, Timage] + En/dedee
Update input embeddings
6: T next + fVLM (IE, ids)
Predict next token
Algorithm 2: Extract Physical Information for Image Tokens
Require: Input æ, T Timage
Ensure: Time (tdags thour) and location (lat, Ing) for each image tokens
1: t xta I
& Extract temporal information from time series input (Equation 7
2: tdays thour t- t//24, t%24
3: lat Ing image + x
Extract spatial information for the entire image.
4: lat, Ing +- cett_center([ (T image: lat imtimage: Ing image)
Compute center coordinates for
patches
23

---

Algorithm 3: Physics-Informed Positional Indexing
Require: T teanty T immage f, lat, Ing
Ensure: Physics-Informed ids
idStezt + sequential indexing
b Assign sequential indices text tokens Equation S > Assign [t, lat, Ing] to image tokens Figure
2: id8image f physics-informed indexing
3: ids + jidstent, ids imase
b ids E 3x3 Fext F7 image
Algorithm 4: Variant- Prequency Sinusoidal Encoding
Require: Position pos
Ensure: Variant-frequency position embedding PE
1: P Ptteet - standard sinusoidal function (Equation 1
2: Pimagee +- variant-frequency sinusoidal function Eqqation 9)
3: PE + PEteeti P'saagel
B.2 Variant-frequency sinusoidal function
This section presents the complete formal definition the variant-frequency sinusoidal function. The model dimensions are partitioned into distinct components: temporal dimensions (first half) and spatial dimensions (latter half). Regarding the temporal dimensions, they combine the encoding of tdy and thour- Similarly, for the spatial dimensions, they combine the latitude embeddings for lal and the longitude embeddings for Ing. This dimensional combination enables simultaneous representation of both temporal and spatial characteristics within the unified model framework. We also visualize the function (Figure 4) taking the dmoded = 128 as an example.
Ldog
2T
dmoded
PP(poo
sin(
is 4i
100004//00 dd
Pday
2
tday
2T
dmodel
(E(pos, = cos(
if 4i + 1
10000 1s/dmodet
Pday
2
thour
2T
dyodeel
PE(por,Ait2) sin(
X
jif 4i + 2 <
1000044/dm 4i/mmmte
Phour
2
thour
2T
dmodel
P
PE(pox,(i+3) cos(
if 4i +3
4000040/dmaad
Phour
2
lat
2T
dmodel
PE = sin(
x
if
< 4i < dmodel
10000 4idmodaad
1/2
Plut
2
lat
2T
dmodel
PEipos, cos(
if
< 4i + < dmodel
100004s/ faoks/dmoee---e 1/2
Piat
2
Ing
2x
dmodel
P PF(poe,tit2) = sin!
x
if
< 4i + 2 S dmodel
1000044/dmmaa 1/2
Ping
2
Ing
2T
dmodel
P PE(pos,itts) coo((
4iddmatd
X
Phour
if
2
< 4i + 3 < dmodel
where for temporal dimensions Pday 366 and Phour 24, while for spatial dimensions, Platitude = 180 and Piongitude. 360. Ldng is the day of the year (ranging from 0 to 365) and thour represents the hour of the day (ranging from 0 to 23). lat the latitude of the image token,
and Ing is the longitude of the image token.
24

---

200
250
300
100
-125
1.90
200
20
50
390
1250000, latitude: longitude: 120.0
shape:
1
shape:
Figure 4: The 64-dimensional positional encoding for the physical variables. Each row represents the embedding vector. The final position encoding will be 128-dimensional by combining them.
C Implementation Details
VLMs training We leverage LLama-Factory [60] for training VLMs, utilizing PyTorch [38] on NVIDIA H800 GPUs. The optimization process employs the Adam optimizer [33] with an initial learning rate of 10 (using the cosine scheduler), batch size of 1, and CrossEntropy loss over training epoch. We provide the code for the reproduction.
implemented using their official repositories, adapted to accommodate our typhoon sequence dataset via modifications to the data loader. Configuration follows original specifications: model dimensions of 768 (One Fits All) and 512 (AutoTimes), with batch size of 128, learning rate of 10-4, and 10 training epochs. For other AI models, we use the publicly available platform Hime-Series-Librar (56) to implement them. The parameters for model dimensions and number of heads are based on their implementation (512) with a batch size of 128 and learning rate of 10-4, training epochs of 30, and patience of 10.
Non-vision models One Fits All [62 and AutoTTimes
32
For domain models, the results are provided by the original paper.
Training Cost We use 4xNVIDIA H800 GPUs to train the models for one epoch. The training time varies significantly across model sizes:
PIPE-3B: 2.1 hours
PIPE-7B: 3.7 hours
PIPE-32B (LoRA
14 rank as 8): 0.7 hours
The PIPE-32B variant achieves substantial time efficiency through LoRA, which reduces trainable parameters while maintaining competitive performance (as shown in Tables9JanddIi This demon- strates an effective balance between model capacity and computational overhead. For baseline AI models (including LLM-based variants like AutoTimes (OPT model)), training completes in hour with a single NVIDIA RTX 4090 GPU.
D Dataset
We provide a comprehensive summary of the Digital Typhoon dataset
22
25

---

Table 4: The detailed information of the Digital Typhoon dataset.
Digital Typhoon dataset
Temporal coverage
1978-2023 (present)
Temporal resolution
one hour
Target satellites
Himawari
Spatial coverage
Western North Pacific basin 5km
Spatial resolution
Image coverage
512x512 pixels (1250km from the center)
Spectral coverage
infrared (others the Website)
Map projection
Azimuthal equal-area projection
Calibration
Recalibration
Data format
HDF5
Best track
Japan Metcorological Agency
Dataset browsing
Digital Typhoon website
Table 5: Multimodal time series forecasting results (leading time is 12h).
Intensity (hPa)
Latitude (*)
Longitude ())
Distance (km)
Models
MAE RMSE
MAE RMSE
MAE
RMSE
MAE
BCCWW--HRES 10
44.972
domain
PanGu 3
44.630
GenCast
40
37.930
TIFS 35
9.061
PatchTST
34
3.917
5.989
0.465
0.615
0.751
0.931
103.818
iTransformer
31
4.004
6.157
0.412
0.558
0.565
0.736
83.174
Crossformer
51
4.257
6.303
0.546
0.726
0.844
1.109
118.748 748
w/o vision
TimeXer 53
5.380
7.911
0.563
0.755
0.713
0.962
108.665
TiDE 9
3.926
6.080
0.416
0.561
0.677
0.850
93.570
One Fits All
62
4.039
6,212
0,420
0.568
0.586
0.759
85.555
AutoTimes 32
4.086
6,220
0.448
0.600
0.692
0.872
97.244
TimesNet 55
4.798
7,220
0.892
1.133
1.796
2.147
230.376
TimeMixer
50
4.227
6.290
0.400
0.533
0.524
0.685
78.569
vision
Original paper
22
12.100
Qwen-2.5-VI.-3B
3.963
6,599
0.371
0.533
0.435
0.610
69.959
PIPE-3B
3.855
6.333
0.359
0.526
0.411
0.587
67.114
E Supplementary Results
E.1
Forecasting Results of More Leading Times
We present additional forecasting analyses this section. First, we list the 12-hour lead-time fore- casting performance of all baseline models (Table 5). Our model demonstrates satt-oofthheaat re- sults across the majority the evaluation metrics. Second, we list the result of the ablation study when the leading time is 12 hours. The consistency between results at different leading times con- firms the robustness of our architectural design, demonstrating that all modules contribute meaning- fully to forecasting accuracy. Finally, we visualize the MAE for the forecasting of pressure, latitude, longitude, and distance across lead times ranging from I to 12 hours (Figure 5). The results con- firm that our model consistently achieves the lowest MAE values at all forecast leading times. This systematic advantage over baseline models highlights the effectiveness of our model in maintaining forecasting precision the leading time increases.
26

---

Table 6: The results of the ablation study (leading time is 12h).
Intensity (hPa)
Latitude (*)
Longitude (?)
Distance (km)
Models
MAE
RMSE
MAE
RMSE
MAE
RMSE
MAE
w/o vision
4.120
6.820
0.372
0.532
0,436
0,616
70.138
w/o 3D indexing (using sequence)
3,936
6.809
0.366
0.535
0.434
0.611
69.382
w/o physics-informed indexing (using 3D)
3.963
6.599
0.371
0.535
0.435
0.610
69,959
w/o negative indexing
4.282
7.017
0.550
0.806
0.869
1,306
124.329
w/o entire sinusoidal function
3.827
6.387
0.364
0.527
0.416
0.590
67,904
w/o varian-frequency sinusoidal function
4.071
6.689
0.370
0.537
0.429
0.604
69.389
PIPE-3B
3.855
6,333
0.359
0.526
0.411
0.587
67.114
Pressure MAB
owen2 PotchTST
itransoom
Crassforer
Timeder
TDEE
Ore Hit All
Auholltere
2
3
12
Lafitude MAB
swanz
Neishtst
mennttome
Cresstamnr
0.3-
Timeder
like
One Pite
Autolilees
2
Longitude MAB
0.8
PRC
qvenz 5-YL
PotchTST
0.6
crossforner
0.4
Onw
Auto limes
TemeMoser
2
3
6
7
10
12
120
Distance MAE
100
gwen2
PotchTST
80
iiransoome
Crassfomer
60
Timeder
40-
TOE
One Riss All
7 Leading time
10
11
12
Figure 5: The performance across leading times ranging from to 12 hours.
E.2 Experiment Statistical Report
The stability of PIPE's forecasting performance is validated through standard deviation analysis across three random seeds, reported in Tabll and Table 8
E.3
Scaling Behavior
To evaluate the impact of model size performance, we conduct experiments across three variants: PIPE-3B, PIPE-7B, and PIPE-32B (with LoRA rank as 8). As demonstrated Tables TO] the largest model, PIPE-32B, yields performance improvements, even when leveraging LoRA.
27

---

Table 7: The mean and the standard deviation of PIPE-3B from three random seeds (leading time is
6 hours)
Intensity (hPa)
Latinue (
Longitude (*)
Distance (km)
Models
MAE
RMSE
MAE
RMSE
MAE
RMSE
MAE
PIPE-38-sced]
1,503
2,940
0.085
0.161
0.095
0.178
16.364
PIPE-3R-seen
1.513
2.946
0.082
0.154
0.094
0.173
16,000
PIPE 3B (B--3B--eedll seeds
1.529
3.059
0.084
0.161
0.097
0.181
16.463
PIPE-3B
1.155 + 0.011
2.981 0.055
0.084
0.001
0.159
0.003
0.095 0.001
0.178
0.003
16.275 + 0.200
Table The mean and the standard deviation of PIPE-3B from three random seeds (leading time is
12 hours).
Intensity (hPa)
Latitie (*)
Longitude (*)
Distance (km)
Models
MAE
RMSE
MAE
RMSE
MAE
RMSE
MAE
PIPE-38-sced1
3.840
6.295
0.362
0.530
0.412
0.590
67.432
PIPE-3B-seed2
3.831
6.281
0.355
0.523
0.405
0.578
66. 402
PIPE-3B seeds
3,893
6.425
0.360
0.527
0.415
0.592
67.506
PIPE-38
3,855 = 0.027
6.333 # 0.065
0.359
0.003
0.526 = 0.003
0.411 = 0.004
0.587 = 0.006
67.114 # 0.050
Table 9: The results of PIPE-3B, PIPE-7B, and PIPE-32B (with LoRA) with the lead time of 6 hours.
Intensity (hPa)
Latitude (?)
Longitude (?)
Distance (km) MAE
Models
MAE
RMSE
MAE
RMSE
MAE
RMSE
PIPE-3B
1.515
2.981
0.084
0.159
0.095
0.178
16.275
PIPE-7B
1.505
2.918
0.088
0.166
0.102
0.184
17.194
PIPE-32B
1.505
2.874
0.079
0.153
0.097
0.182
15.980
E.4 Forecasting Results on Different Grades
We experiment to investigate the performance of our method on different grades of typhoons. We split the dataset based on the grade (from grade to grade 6). Grade 2 represents tropical depres- sions, which are weaker cyclones and not classified tropical cyclones. Grades 3, 4, and 5 represent tropical cyclones, with Grade 5 being the most intense. Grade 6 represents a cyclone with differ- ent structural system from tropical cyclones. Then we train models on different sub-datasets. The results are shown in Table
Track Forecasting vs. Pressure Forecasting Tropical cyclones (Grades 3, 4, 5) demonstrate bet- ter track forecasting performance compared to non-tropical cyclones (Grades 2, 6). This is likely due to the more stable and predictable system of tropical cyclones, which facilitates track forecasting. However, pressure forecasting is more challenging for tropical cyclones due higher variability and complexity. contrast, non-troe con-tropical cyclones (Grade 2 and Grade 6) exhibit more predictable pressure patterns but greater difficulty in track forecasting, This is especially pronounced before the cyclone structure forms (Grade 2) and after it breaks (Grade 6).
Impact of Intensity Forecasting As the intensity of tropical cyclones increases (Grades 3, 4, 5), pressure forecasting becomes progressively more challenging. However, the difficulty of track forecasting remains relatively consistent across these grades.
Impact of Data Volume Models trained on all grades perform better in track forecasting compared to models trained on individual grades. This result indicates that larger volume of training data improves track forecasting accuracy across different cyclone intensities.
F.5 Experiment on the Offset Indexing
In addition to negative indexing, we investigate alternative approach called offset indexing, where all image tokens are assigned an offset equal the maximum length of the textual tokens address conflicts between the physical information of vision tokens and the order information of text tokens.
28

---

Table 10: The results of PIPE-3B, PIPE-7B, and PIPE-32B (with LoRA) with the leading time of
12 hours.
Intensity (hPa)
Latitude ())
Longitude (9)
Distance (km)
Models
MAE
RMSE
MAE
RMSE
MAE
RMSE
MAE
PIPE-3B
3.855
6.333
0.359
0.526
0.411
0.587
67.114
PIPE-7B
3.861
6.325
0.371
0.540
0.435
0.609
69.933
PIPE-32B
3.695
6.029
0.342
0.510
0.423
0.610
66.725
Table 11: The results of the experiment across different grades with the lead time of hours.
Intensity (hPa) MAE
Latitude (9) MAE
Longitude (9) MAE
Distance (km)
Models
RMSE
RMSE
RMSE
MAE
Grade 2
0.722
1.227
0.108
0.206
0.138
0.245
22.029
Grade 3
0.997
1.772
0.097
0.169
0.115
0.200
18.902
Grade 4
1.656
2.732
0.099
0.156
0.122
0.188
19.310
Grade 5
2.707
4.556
0.090
0.163
0.116
0.203
18.418
Grade 6
1.074
1.830
0.208
0.385
0.283
0.488
37.193
The results of this experiment are presented in Table 12 We find that although offset indexing offers some performance improvement, it falls short compared to PIPE. Our choice of negative indexing remains an intuitive and efficient method for effectively separating different types of tokens.
E.6
Showcase
We present prediction showcase (Figure 6) to compare our method with the methods that remove satellite imagery and PIPE on the Typhoon Phanfone. PIPE achieves more accurate track forecasting and intensity forecasting.
Typhoon Comparison
Central Prediction
15PN
Ground
1000
13.5
PIPE-B
without satellite imagery without
Pressura (hFa)
10.5
97N
980
7.5%M
960
115'E
120'E
125'E
130'E
135'E
140'E
145"S
201912/00 1400
20012/23 0210
20992/23 LAND
Figure 6: The results of Typhoon Phanfone comparison between PIPE, removing satellite images, and removing PIPE. The leading time is 12 hours and the time gap between neighbouring dots is 12 hours.
E.7 Attention Analysis
We compare the attention from the penultimate layer (Figure 7) with averaging across the head dimension of PIPE-3B and Qwen2.5- It reveals distinct attention patterns. --ee2.5--LL33B exhibits an obvious bias toward the initial tokens of the historical time series, as evidenced by an obvious vertical line at 800th input tokens (()) & (f)). In contrast, our PIPE model allocates greater attention to both the image tokens and the historical time series tokens. Notably, PIPE's attention on image patches is concentrated the typhoon region (c.g., central cloud structure), whereas
29

---

Table 12: The results of offset indexing.
Intensity (hPa) MAE
Latitude (*) RMSE
Longitude ())
Distance (km)
Models
RMSE
MAE
MAE
RMSE
MAE
no-indexing
1,961
3,926
0.206
0.360
0.388
0.674
53,548
offset indexing
1.815
3.762
0.188
0.303
0.229
0.385
30.157
negative indexing (PIPE)
1.515
2.981
0.084
0.159
0.095
0.178
16.275
Qwen2.5-VL-3B's attention appears diffuse and unstructured across the image. These differences in attention mechanisms likely contribute to PIPE's better forecasting accuracy.
055
400-
400
100
0.0
200
20
400
800
002
0.4
2000
200
02
52
3800
203
402
600
$0.0
1006
33000
Tooa
700
400
600
830
3270
1200
4eaimap
1s.12ap
Figure 7: Visualization of attention (normalized to 0-1 in each step) from the penultimate layer of the PIPE model (top) and the Qwen2.5-VL--3B model (bottom), averaged across attention heads. (a) & (e) The entire attention matrix. (b) & (f) The attention matrix of predicted tokens' attention on the input tokens, including image tokens and history time series tokens. (c) (g) Attention of predicted tokens on the first input image. (d) & (h) Attention of predicted tokens the last input image.
We also compare the attention using Attention Rollout I (Figure 8) with averaging across the head dimension of PIPE-3B and Swee2.55VL--B. also demonstrates that our model allocates more reasonable attention to image tokens and historical time series tokens. Furthermore, our model's attention on image patches is focused specifically on the typhoon region, whereas Qwen2.5-VL. 3B's attention appears biased across the image.
30

---

30
200
62
aaagee
30
100
h
220
100
Figure 8: Visualization of attention (normalized to 0-1 in each step) using Attention Rollout of the PIPE model (top) and the Qwen2.5-VL-33 model (bottom), averaged across attention heads. (a) & (e) The entire attention matrix. (b) & (f) The attention matrix of predicted tokens' attention on the input tokens, including image tokens and history time series tokens. (c) & (g) Attention of predicted tokens on the first input image. (d) & (h) Attention of predicted tokens on the last input image.
F Broader Impact
We introduce novel multimodal time series forecasting task that integrates satellite imagery with temporal data for capturing complex spatio-temporal dependencies. This approach leverages the complementary strengths of temporal time series data and spatially rich visual inputs, enabling mod- els to go beyond the limitations of traditional univariate, multivariate, single-modality methods. To address the inherent challenges of integrating satellite imagery into time series forecasting, we propose physics-informed positional encoding. This technique incorporates physical information derived from satellite data, such as geospatial coordinates, to enhance the model's s ability to reason about spatial and temporal dependencies. This innovation is particularly relevant for applications where visual inputs carry critical physical context, including climate modeling, urban planning, and agricultural forecasting. The broader impact of this work lies in its ability bridge the gap between traditional forecasting methods and real-world complexities that often include spatial and physical components. By incorporating satellite imagery and physics-informed informed encoding, this method has potential benefits across a wide range of scientific and practical domains.
31


Reviews:
[{"review_id": "yjoGZrp0cL", "reviewer": "Reviewer_NJNg", "summary": "This paper introduces PIPE (Physics-Informed Position Encoding), a framework for multimodal time series forecasting, with a focus on integrating satellite imagery and numerical data for tasks such as typhoon prediction. Unlike existing models that either ignore visual data or only capture pixel-level semantics, PIPE incorporates physical metadata (timestamps, latitude, longitude) directly into the position encoding of vision-language models (VLMs). \nPIPE is evaluated on Typhoon dataset and achieves state-of-the-art forecasting accuracy, notably improving typhoon intensity and track predictions compared to both AI and domain-specific baselines.", "questions": "Please see weakness, if you can address them", "limitations": "One paragraph on limitations of the work should be included", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths\nIntroducing PIPE which explicitly embeds physical attributes—like timestamps, latitude, and longitude—into both the positional indexing and encoding schemes of a vision-language forecasting model. This is beyond traditional positional encoding (which only preserves token order) by incorporating global, real-world geospatial and temporal information directly into the model’s structure. This aligns satellite imagery with time-series data, enabling the model to understand not just visual patterns but also their physical context, crucial for accurate climate forecasting. Authors have performed rigorous experiments to establish their proposals.\n\nWeaknesses\nIt has been tested only on one domain, experiments could be performed on other applications such as crop monitoring, climate change, etc.\n\nAlthough, the title of the paper is physics informed…, it does not modal physics laws, constraints etc.\n\nWhile PIPE innovatively encodes timestamps and geo-coordinates (latitude & longitude), it does not incorporate other rich physical attributes (like atmospheric pressure fields, wind vectors, or physical constraints from fluid dynamics).\n\nThe authors themselves note future work should integrate broader physical laws and constraints to better model real-world dynamics.\n\nWhile PIPE improves short (6–12 hour) forecasts, I am not sure how does it analyze very long-term forecasting performance (e.g., multiple days, weeks ahead) where errors compound and encoding choices might have more dramatic effects.\n\nAuthors could have included the comparison between standard positional encodings and variant -frequency positional encodings.\n\nFew observations:\n\nThe statement in the abstract: “However, existing multimodal approaches primarily ….., leaving the visual data in existing time series datasets untouched.” The integration of time series and vision (multimodal) data is discussed in the following works: Paper 1, Paper 2. But the wording is overly absolute.\n\nIn Figure 1, the axes and legend require clarification. The units for intensity are not explicitly labelled (e.g., hPa)\nIn Figures 4 and 5, the x-axis, y-axis, and legend are not properly visible. \nAlgorithm 1 and related descriptions are insightful, but formatting inconsistencies (e.g., indentation of bullet points, alignment of variables) make it hard to follow. Please refine for clarity.", "was_revised": true, "final_justification": "The authors have given satisfactory reply to all my questions and given additional results as well. I have increased their ratings.", "rebuttal": "Dear Reviewer [NJNg],\n\nThank you for your efforts and positive feedback. \n\n> Weakness #1: It has been tested only on one domain, experiments could be performed on other applications such as crop monitoring, climate change, etc.\n\nThank you for your suggestion. To demonstrate the generalizability of our method, we added two additional experiments targeting different application domains: one focuses on typhoon forecasting in another region (the Australian (AU) region) [1]; the other uses the T31TFM-16 dataset for crop monitoring [2]. The first experiment confirms the robustness of our method in typhoon forecasting, while the second demonstrates its applicability to a different domain, showcasing its broader generalization capability.\n\n**1. AU typhoon experiment:**  \n  * Setting: The settings remain the same as PIPE. The focus is on typhoon forecasting, a regression task, which is the core of our paper.\n  * Result:\n|Models|Intensity MAE|Intensity RMSE|Latitude MAE|Latitude RMSE|Longitude MAE|Longitude RMSE|Distance MAE|\n|---|---|---|---|---|---|---|---|\n|Qwen-2.5-VL-3B (after tuning)|1.399|2.806|0.098|0.201|0.121|0.211|19.718|\n|PIPE-3B (Zero-shot)|1.361|2.773|0.094|0.171|0.118|0.206|19.090|\n|PIPE-3B (after tuning)|1.352|2.558|0.093|0.163|0.116|0.202|18.835|\n  * Analysis: The results show that our method can be generalized to different regions for typhoon forecasting. Even the model trained on the West Pacific region can perform well in the new dataset (zero-shot). For instance, compared to Qwen-2.5-VL-3B (after tuning), PIPE-3B (Zero-shot) achieved a lower intensity RMSE (2.773 vs. 2.806) and a smaller distance error (19.090 vs. 19.718). Additionally, PIPE-3B (after tuning) outperformed both models, achieving the best overall performance.\n\n**2. Crop monitoring experiment:**\n\n  * Setting: We adopt our method to the semantic segmentation task. The training data and test data were split by the original dataset, with a ratio of 0.85:0.15. We calculate latitude and longitude from the given UTM information provided in the dataset. We use the ViT model to encode the satellite image patches and incorporate each patch’s physical information. The focus is on crop monitoring, a segmentation task. Number of epochs: 50. lr: 1e-4. Optimizer: Adam.\n\n  * Result:\n|Models|IoU|Accuracy|\n|---|---|---|\n|ViT PE|67.52|80.61|\n|3D PE|67.27|80.43|\n|PIPE|69.21|81.81|\n\n  * Analysis: The results show that our method achieves the best result (69.21% and 81.81%) than standard ViT PE (67.52% and 80.61%)  and 3D PE (67.27% and 80.43%). The experiment shows that our method can be generalized to other domains by incorporating global physical information.\n\n> Weakness #2: Although, the title of the paper is physics informed…, it does not modal physics laws, constraints etc. While PIPE innovatively encodes timestamps and geo-coordinates (latitude & longitude), it does not incorporate other rich physical attributes (like atmospheric pressure fields, wind vectors, or physical constraints from fluid dynamics). The authors themselves note future work should integrate broader physical laws and constraints to better model real-world dynamics.\n\nThank you for your advice. \n\n1. We would like to respectfully emphasize that, to the best of our knowledge, our method is the first to incorporate global physical information in a way that enables VLMs to effectively capture the visual features of satellite imagery, marking a significant step forward in this emerging research direction.\nOur core contribution lies in the proposed **physics-informed position encoding**, which leverages spatiotemporal metadata (e.g., timestamps, latitude, and longitude) to inject physical context into the model. Broadening the scope to include additional physical constraints or features beyond positional encoding would have introduced substantial complexity and risked diluting the focus of this paper. Instead, we deliberately centered our efforts on thoroughly evaluating the effectiveness of our position encoding approach through comprehensive experiments and analysis. We recognize the importance of integrating broader physical laws and plan to pursue this promising direction in future work.\n\n2. Furthermore, we have explicitly acknowledged this limitation and highlighted the integration of additional physical attributes and constraints (e.g., atmospheric pressure, wind vectors, and fluid dynamics laws) as an important direction for future work (line 643 in the original manuscript). In line with the review guidelines, We believe that being up front about the limitations should be rewarded rather than punished.\n\n> Weakness #3: While PIPE improves short (6–12 hour) forecasts, I am not sure how does it analyze very long-term forecasting performance (e.g., multiple days, weeks ahead) where errors compound and encoding choices might have more dramatic effects.\n \nThank you for your feedback.\n \n1. **Short-term forecasting is of critical importance** for timely decision-making, as typhoon events typically span only a few days. For example, the maximum forecasting length is limited to 12 hours in the dataset baseline used in our study and limited to 1 hour in [3], which aligns with the practical need for short-term predictions. \n2. We have also included the extended forecasting horizons for VLMs in the Limitations section (line 642). Addressing the challenge of long-term forecasting will be one of the key focuses of future work, particularly addressing the limitation of current VLMs to handle more input images. However, the **primary goal of this paper is to explore whether PIPE is effective** in incorporating additional physical information into forecasting models. We believe that our experimental results successfully demonstrate the validity of our methods.\n\n> Weakness #4: Authors could have included the comparison between standard positional encodings and variant -frequency positional encodings.\n\nThank you for the valuable advice. We added the experiment as suggested:\n\n|Models|Intensity MAE|Intensity RMSE|Latitude MAE|Latitude RMSE|Longitude MAE|Longitude RMSE|Distance MAE|\n|---|---|---|---|---|---|---|---|\n|stantard PE|1.617|3.231|0.087|0.162|0.103|0.187|17.129|\n|only variant-frequency PE|1.581|2.994|0.087|0.161|0.102|0.186|17.117|\n\nThe results show that the variant-frequency component can improve the performance over the standard positional encodings, since it encodes the physical attributes of the variables.\n\n> Observation #1: The statement in the abstract: “However, existing multimodal approaches primarily ….., leaving the visual data in existing time series datasets untouched.” The integration of time series and vision (multimodal) data is discussed in the following works: Paper 1, Paper 2. But the wording is overly absolute.\n\nThank you for your advice. We agree that the current phrasing is not the most appropriate, and we will revise the statement to:  ‘leaving the visual data in existing time series datasets **underexplored**’.\nWe would like to respectfully state that to our best knowledge, we are the first to integrate global physical information into position encoding to capture visual features. The existing methods focus on the incorporation of textual data in time series forecasting. \nWe notice that you left two placeholders for existing works toward this research direction. We would greatly appreciate it if you could share these references with us to further improve our work. \n\n> Observation #2: In Figure 1, the axes and legend require clarification. The units for intensity are not explicitly labelled (e.g., hPa) In Figures 4 and 5, the x-axis, y-axis, and legend are not properly visible. Algorithm 1 and related descriptions are insightful, but formatting inconsistencies (e.g., indentation of bullet points, alignment of variables) make it hard to follow. Please refine for clarity.\n\nThank you for the positive feedback on our algorithm and the advice to improve the presentation quality.\nIn Figure 1, the x-axis is the longitude and the y-axis is the latitude. We will add hPa to the intensity. \nIn Figures 4 and 5, we will increase the font size of the x-axis, y-axis, and legend.\nWe will revise the format of Algorithm 1.\n\n> Limitations #1: One paragraph on limitations of the work should be included\n\nWe have included the limitation in Appendix G (line 638). We will move them into the main section.\n\n\\\nFinally, we want to thank you once again for your thoughtful comments. We look forward to your feedback and we kindly ask that you consider these clarifications in your final grading.\n\nWarm regards,\n\nThe Authors\n\n\\\n[1] Kitamoto, Asanobu, Erwan Dzik, and Gaspar Faure. \"Machine Learning for the Digital Typhoon Dataset: Extensions to Multiple Basins and New Developments in Representations and Tasks.\" arXiv preprint arXiv:2411.16421 (2024).\n\n[2] Tarasiou, Michail, Riza Alp Güler, and Stefanos Zafeiriou. \"Context-self contrastive pretraining for crop type semantic segmentation.\" IEEE Transactions on Geoscience and Remote Sensing 60 (2022): 1-17.\n\n[3] Zhu, Jiakai, and Jianhua Dai. \"A rain-type adaptive optical flow method and its application in tropical cyclone rainfall nowcasting.\" Frontiers of Earth Science 16.2 (2022): 248-264."}, {"review_id": "l61gmt7G04", "reviewer": "Reviewer_8J4Q", "summary": "This paper proposes PIPE (Physics-Informed Positional Encoding), a multimodal time-series forecasting framework that integrates visual information using a VLM. The key contribution is its physics-informed positional encoding, which embeds real-world metadata (latitude, longitude, and timestamp) into visual token positions, enabling better spatiotemporal alignment across inputs. A variant-frequency sinusoidal function further captures the periodic nature of physical variables. PIPE is evaluated on typhoon forecasting, showing that incorporating physical context improves multimodal prediction.", "questions": "See the comments above. \n\nAdditionally, why does the Figure 1 show two distinct points with the same timestamp (2018/10/29 00:00) but different latitude, longitude, and intensity values?", "limitations": "Have the authors adequately addressed the limitations and potential negative societal impact of their work? If so, simply leave “yes”; if not, please include constructive suggestions for improvement. In general, authors should be rewarded rather than punished for being up front about the limitations of their work and any potential negative societal impact. You are encouraged to think through whether any critical points are missing and provide these as feedback for the authors.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths\n1.\tThe paper proposes a lightweight method to embed physical information into VLMs, allowing the model to encode shared global physical knowledges in a simple way. This approach has potential benefits for a wide range of transformer-based models in time-series prediction tasks.\n2.\tThe framework makes effective use of historical physical metadata (e.g., latitude, longitude, and time): once as explicit values in the textual input, and again through physics-informed positional indexing applied to visual tokens. \n3.\tThe application to typhoon forecasting is highly relevant and impactful, with potential real-world benefits for disaster preparedness and evacuation planning.\n4.\tThe paper is clearly structured, and easy to follow.\n\nWeakness\n1.\tThe first claimed contribution may be overstated. While the paper introduces a novel positional encoding scheme, the overall architecture closely follows the standard structure of VLMs for multimodal input processing.\n2.\tThe comparison against baselines is potentially unfair. Most baseline models do not incorporate visual data, except for Qwen-2.5-VL-3B. Since the performance gains over Qwen-2.5-VL-3B are relatively small, the strength of the contribution may be overstated without broader comparison to other vision-enabled baselines.\n3.\tWhile the proposed method is demonstrated on typhoon forecasting, other real-world applications could benefit from embedding physical information. If the results in the typhoon domain cannot be reliably reproduced, it would be valuable for the authors to evaluate the method in additional domains to support its generalizability.\n4.\tMinor typographical issue: in line 141, there should be a space between “encoding” and “PIPE (PIPE).”", "was_revised": true, "final_justification": "The authors have provided more results, which have partially addressed my concerns.", "rebuttal": "Dear Reviewer [8J4Q],\n\nThank you for your time and feedback.\n\n> Weakness #1: The first claimed contribution may be overstated. While the paper introduces a novel positional encoding scheme, the overall architecture closely follows the standard structure of VLMs for multimodal input processing.\n\nThank you for your advice on the writing. We will revise the statement as suggested to ‘We propose a novel positional encoding **scheme** that …’ to ensure the wording is accurate and not overstated.\n\nMeanwhile, we would like to respectfully clarify that the essence of our first contribution remains strong and valid. To our best knowledge, we are the first to integrate global physical information into position encoding to capture visual features.  While existing approaches primarily focus on incorporating textual or tabular data in time series forecasting, our method opens a new direction by enabling VLMs to effectively leverage physical context through position encoding.\n\n> Weakness #2: The comparison against baselines is potentially unfair. Most baseline models do not incorporate visual data, except for Qwen-2.5-VL-3B. Since the performance gains over Qwen-2.5-VL-3B are relatively small, the strength of the contribution may be overstated without broader comparison to other vision-enabled baselines.\n\nThank you for your comment. We have tried our best to cover the majority of available baselines, including domain-specific models and machine learning models (both vision-based and non-vision-based). This comprehensive evaluation is also appreciated by Reviewer [NJNg], who noted that we “have performed rigorous experiments.” However, due to the limitations of existing methods, many of them did not incorporate vision data for time series forecasting, which further highlights the novelty of our approach.\n\nFurthermore, we would like to respectfully argue that:\n1. Besides Qwen-2.5-VL-3B, we also have included other benchmark results in the original manuscript from the dataset benchmarking, which incorporates visual data. For instance, in **Table 4**, the 12-hour result is included for comparison (only the 12-hour result is available in the original benchmarking). \n2. To further enhance the vision-enabled baselines and demonstrate the effectiveness of our method, besides the open-source Qwen-2.5-VL-3B, we added one additional experiment on closed-source MLLM: gemini-2.5-flash, which also incorporates visual data.\n  * Experiment setting: the input remains the same settings as Qwen and PIPE.\n  * Result: \n|Models|Intensity MAE|Intensity RMSE|Latitude MAE|Latitude RMSE|Longitude MAE|Longitude RMSE|Distance MAE|\n|---|---|---|---|---|---|---|---|\n|gemini-2.5-flash|1.924|3.654|0.174|0.254|0.237|0.798|36.006|\n|PIPE|1.515|2.981|0.084|0.159|0.095|0.178|16.275|\n  * Analysis: The results show that our method can be **more effective** in integrating satellite visual features than existing multimodal methods.\n\n3. We believe that integrating visual data represents our unique contribution, addressing a significant research gap in multimodal forecasting methods. The existing approaches are limited since they mainly align numeric (and text) data. We respectfully believe this should be seen as a strength rather than a weakness.\n4. Regarding the gain over Qwen-2.5-VL-3B, our method achieved: 6.7% (intensity MAE)，8.4% (intensity RMSE),  3.5% (lat MAE), 1.9% (lat RMSE), 8.4%(lng MAE), 5.1%(lng RMSE), and 5.2% (distance). We believe that these results are able to demonstrate the effectiveness of our proposed method.\n\n> Weakness #3: While the proposed method is demonstrated on typhoon forecasting, other real-world applications could benefit from embedding physical information. If the results in the typhoon domain cannot be reliably reproduced, it would be valuable for the authors to evaluate the method in additional domains to support its generalizability.\n\nThank you for your suggestion. To demonstrate the generalizability of our method, we added two additional experiments targeting different application domains: one focuses on typhoon forecasting in another region (the Australian (AU) region) [1]; the other uses the T31TFM-16 dataset for crop monitoring [2]. The first experiment confirms the robustness of our method in typhoon forecasting, while the second demonstrates its applicability to a different domain, showcasing its broader generalization capability.\n\n1. AU typhoon experiment:  \n  * Settings: The settings remain the same as PIPE. The focus is on typhoon forecasting, a regression task, which is the core of our paper.\n  * Result:\n|Models|Intensity MAE|Intensity RMSE|Latitude MAE|Latitude RMSE|Longitude MAE|Longitude RMSE|Distance MAE|\n|---|---|---|---|---|---|---|---|\n|Qwen-2.5-VL-3B (after tuning)|1.399|2.806|0.098|0.201|0.121|0.211|19.718|\n|PIPE-3B (Zero-shot)|1.361|2.773|0.094|0.171|0.118|0.206|19.090|\n|PIPE-3B (after tuning)|1.352|2.558|0.093|0.163|0.116|0.202|18.835|\n\n  * Analysis:\nThe results show that our method can be generalized to different regions for typhoon forecasting. Even the model trained on the West Pacific region can perform well in the new dataset (zero-shot). For instance, compared to Qwen-2.5-VL-3B (after tuning), PIPE-3B (Zero-shot) achieved a lower intensity RMSE (2.773 vs. 2.806) and a smaller distance error (19.090 vs. 19.718). Additionally, PIPE-3B (after tuning) outperformed both models, achieving the best overall performance.\n\n2. Crop monitoring experiment:\n  * Setting: We adopt our method to the semantic segmentation task. The training data and test data were split by the original dataset, with a ratio of 0.85:0.15. We calculate latitude and longitude from the given UTM information provided in the dataset. We use the ViT model to encode the satellite image patches and incorporate each patch’s physical information. The focus is on crop monitoring, a segmentation task. Number of epochs: 50. lr: 1e-4. Optimizer: Adam.\n\n  * Result:\n|Models|IoU|Accuracy|\n|---|---|---|\n|ViT PE|67.52|80.61|\n|3D PE|67.27|80.43|\n|PIPE|69.21|81.81|\n\n  * Analysis: The results show that our method achieves the best result (69.21% and 81.81%) than standard ViT PE (67.52% and 80.61%)  and 3D PE (67.27% and 80.43%). The experiment shows that our method can be generalized to other domains by incorporating global physical information.\n\n> Weakness #4: Minor typographical issue: in line 141, there should be a space between “encoding” and “PIPE (PIPE).”\n>\n> Question #1: why does the Figure 1 show two distinct points with the same timestamp (2018/10/29 00:00) but different latitude, longitude, and intensity values?\n\nThank you for your feedback. We will correct the typos.\n\n\\\nFinally, we want to thank you once again for your thoughtful comments. We look forward to your feedback and we kindly ask that you consider these clarifications in your final grading.\n\nWarm regards,\n\nThe Authors\n\n\\\n[1] Kitamoto, Asanobu, Erwan Dzik, and Gaspar Faure. \"Machine Learning for the Digital Typhoon Dataset: Extensions to Multiple Basins and New Developments in Representations and Tasks.\" arXiv preprint arXiv:2411.16421 (2024).\n\n[2] Tarasiou, Michail, Riza Alp Güler, and Stefanos Zafeiriou. \"Context-self contrastive pretraining for crop type semantic segmentation.\" IEEE Transactions on Geoscience and Remote Sensing 60 (2022): 1-17."}, {"review_id": "pd9hbdjC9m", "reviewer": "Reviewer_hPaZ", "summary": "To address the challenge of effectively integrating images and structured data in multimodal time series forecasting, this paper proposes a Physics-Informed Position Encoding (PIPE) method. PIPE aims to better extract the underlying time and position information embedded in satellite images and enhance the alignment between visual data and time series. The method is applied to complex meteorological scenarios such as typhoon intensity and trajectory forecasting.", "questions": "1. Can PIPE be extended to other domains such as remote sensing in agriculture, urban heat island analysis, or sea ice prediction? Have similar tasks been attempted?\n\n2. Have you conducted experiments to evaluate the performance of PIPE across different typhoon intensity levels? I suppose the performance may vary depending on the severity of the typhoon.\n\n3. Apart from using negative indices, have you considered other mechanisms to separate visual and textual tokens (e.g., using token type embeddings)?", "limitations": "It would be helpful to further elaborate on the applicability of the proposed model—whether it is limited to weather-related tasks or potentially effective across a broader range of multimodal applications.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "1. Advantages:\n\n(1) This paper proposes the PIPE method as a targeted solution for effectively integrating sequential data and satellite imagery. While it is applied to typhoon prediction in this work, the approach has significant practical relevance.\n\n(2) The proposed physics-informed positional indexing method distinguishes image patches from sequential data tokens using negative indices—a novel and elegant design that effectively addresses index conflicts.\n\n(3) PIPE achieves state-of-the-art performance in both deep learning-based forecasting and climate modeling tasks.\n\n2. Weaknesses :\n\n(1) The main contributions of the paper focus on incorporating physical information through positional indexing and encoding at the model input level. Although the paper claims to be physics-informed, the physical guidance is limited to input encoding and does not extend to more meaningful components such as physical knowledge or physics-guided loss functions or constraints on model behavior.\n\n(2) The title of this paper is for alignment of satellite images and time series, but the experiments are restricted to the domain of typhoon forecasting and only one testing dataset, which cannot fully support the main claim and limits the generalization of the proposed method. \n\n(3) The paper does not investigate how PIPE performs across different categories of typhoon intensity. Since typhoons are typically classified into six levels, conducting experiments to analyze the model’s performance across these different intensity levels would provide deeper insights into its robustness and practical utility.\n\n(4) If the evaluation remains limited to a single typhoon dataset, I suggest explicitly mentioning \"typhoon\" in the title to reflect the task more accurately. Moreover, if the focus is indeed on typhoon prediction, the method should be validated on additional typhoon datasets and compared with more typhoon-specific baselines.", "was_revised": true, "final_justification": "The authors promptly and thoroughly reply to my concerns, mainly solving the issues about the title specific for typhoon, more validation datasets and special validation on key components. Therefore, I prefer to increase my original rating to a positive one. However, I could not agree that \"being up front about the limitations should be rewarded rather than punished“. It depends on whether the limitation is a key problem. I think for physical information, position encoding only occupies a very small part. That's why I could not increase more.", "rebuttal": "Dear Reviewer [hPaZ],\n\nThank you for your efforts and comments.\n\n> Weakness #1: The main contributions of the paper focus on ... or constraints on model behavior.\n\nThank you for your advice. \n\n1. We would like to respectfully emphasize that, to the best of our knowledge, our method is the first to incorporate global physical information in a way that enables VLMs to effectively capture the visual features of satellite imagery, marking a significant step forward in this emerging research direction.\n\n    Our core contribution lies in the proposed **physics-informed position encoding**, which leverages spatiotemporal metadata (e.g., timestamps, latitude, and longitude) to inject physical context into the model. Broadening the scope to include additional physical constraints or features beyond positional encoding would have introduced substantial complexity and risked diluting the focus of this paper. Instead, we deliberately centered our efforts on thoroughly evaluating the effectiveness of our position encoding approach through comprehensive experiments and analysis. We recognize the importance of integrating broader physical laws and plan to pursue this promising direction in future work.\n\n2. Furthermore, we have explicitly acknowledged this limitation and highlighted the integration of additional components (e.g., physical knowledge, physics-guided loss functions, or constraints on model behavior) as an important direction for future work (line 643 in the original manuscript). In line with the review guidelines, We believe that being up front about the limitations should be rewarded rather than punished.\n\n> Weakness #2: The title of this paper is for alignment of satellite images and time series, ... limits the generalization of the proposed method.\n>\n> Weakness #4: If the evaluation remains limited to a single typhoon dataset, I suggest explicitly mentioning \"typhoon\" in the title ... more typhoon-specific baselines.\n>\n> Question #1: Can PIPE be extended to other domains ... Have similar tasks been attempted?\n>\n> Limitation #1: It would be helpful to further elaborate on the applicability ... a broader range of multimodal applications.\n\nThank you for your suggestion. To demonstrate the generalizability of our method, we added two additional experiments targeting different application domains: one focuses on typhoon forecasting in another region (the Australian (AU) region) [1]; the other uses the T31TFM-16 dataset for crop monitoring [2]. The first experiment confirms the robustness of our method in typhoon forecasting, while the second demonstrates its applicability to a different domain, showcasing its broader generalization capability.\n\nThough we validate the generalizability of our method by adding two additional experiments, we also would like to **emphasize typhoon forecasting in the title**, as you suggested. The new title will be ‘PIPE: Physics-Informed Position Encoding for Alignment of Satellite Images and Time Series of Typhoon Forecasting’\n\n1. AU typhoon experiment:  \n  * Settings: The settings remain the same as PIPE. The focus is also on typhoon forecasting, a regression task, which is the core of our paper.\n  * Result:\n|Models|Intensity MAE|Intensity RMSE|Latitude MAE|Latitude RMSE|Longitude MAE|Longitude RMSE|Distance MAE|\n|---|---|---|---|---|---|---|---|\n|Qwen-2.5-VL-3B (after tuning)|1.399|2.806|0.098|0.201|0.121|0.211|19.718|\n|PIPE-3B (Zero-shot)|1.361|2.773|0.094|0.171|0.118|0.206|19.090|\n|PIPE-3B (after tuning)|1.352|2.558|0.093|0.163|0.116|0.202|18.835|\n\n  * Analysis:\nThe results show that our method can be generalized to different regions for typhoon forecasting. Even the model trained on the West Pacific region can perform well in the new dataset (zero-shot). For instance, compared to Qwen-2.5-VL-3B (after tuning), PIPE-3B (Zero-shot) achieved a lower intensity RMSE (2.773 vs. 2.806) and a smaller distance error (19.090 vs. 19.718). Additionally, PIPE-3B (after tuning) outperformed both models, achieving the best overall performance.\n\n2. Crop monitoring experiment:\n  * Setting: We adopt our method to the semantic segmentation task. The training data and test data were split by the original dataset, with a ratio of 0.85:0.15. We calculate latitude and longitude from the given UTM information provided in the dataset. We use the ViT model to encode the satellite image patches and incorporate each patch’s physical information. The focus is on crop monitoring, a segmentation task. Number of epochs: 50. lr: 1e-4. Optimizer: Adam.\n\n  * Result:\n|Models|IoU|Accuracy|\n|---|---|---|\n|ViT PE|67.52|80.61|\n|3D PE|67.27|80.43|\n|PIPE|69.21|81.81|\n\n  * Analysis: The results show that our method achieves the best result (69.21% and 81.81%) than standard ViT PE (67.52% and 80.61%)  and 3D PE (67.27% and 80.43%). The experiment shows that our method can be generalized to other domains by incorporating global physical information.\n\n> Weakness #3:The paper does not investigate how PIPE performs across different categories of typhoon intensity. ... practical utility.\n>\n> Question #2: Have you conducted experiments ... depending on the severity of the typhoon.\n\nThank you for your suggestion. We added the experiments on different intensity levels.\n\n  * Experiment settings:\nThe settings remain the same as the experiment in the paper.\n\nAfter splitting, the dataset begins from Grade 2, and the distribution of samples is as follows:\nGrade 2: 18027.\nGrade 3: 17193.\nGrade 4: 8328.\nGrade 5: 10649.\nGrade 6: 15558.\n\n  * Result:\n|Grades|Intensity MAE|Intensity RMSE|Latitude MAE|Latitude RMSE|Longitude MAE|Longitude RMSE|Distance MAE|\n|---|---|---|---|---|---|---|---|\n|Grade 2|0.722|1.227|0.108|0.206|0.138|0.245|22.029|\n|Grade 3|0.997|1.772|0.097|0.169|0.115|0.200|18.902|\n|Grade 4|1.656|2.732|0.099|0.156|0.122|0.188|19.310|\n|Grade 5|2.707|4.556|0.090|0.163|0.116|0.203|18.418|\n|Grade 6|1.074|1.830|0.208|0.385|0.283|0.488|37.193|\n|All|1.515|2.981|0.084|0.159|0.095|0.178|16.275|\n\n  * Analysis:\n    * Definition of grades (helps to understand the results):\n\n      * Grades 3, 4, and 5 represent tropical cyclones, with Grade 5 being the most intense (not Grade 6).\n\n      * Grade 2 represents tropical depressions, which are weaker cyclones and not classified as tropical cyclones.\n\n      * Grade 6 represents a cyclone with a different structural system from tropical cyclones (not by tense). Typhoons typically transit to Grade 6 as their structure changes and then finish, instead of reverting to lower grades like 5, 4, 3, or 2.\n  * Results analysis:\n    * 1. Track Forecasting vs. Pressure Forecasting:\n      * Tropical cyclones (Grades 3, 4, 5) demonstrate better track forecasting performance compared to non-tropical cyclones (Grades 2, 6). This is likely due to the more stable and predictable system of tropical cyclones, which facilitates track forecasting. However, pressure forecasting is more challenging for tropical cyclones due to higher variability and complexity.\n      * In contrast, non-tropical cyclones (Grade 2 and Grade 6) exhibit more predictable pressure patterns but greater difficulty in track forecasting. This is especially pronounced before the cyclone structure forms (Grade 2) and after it breaks (Grade 6).\n    * 2. Impact of Intensity on Forecasting:\n      * As the intensity of tropical cyclones increases (Grades 3, 4, 5), pressure forecasting becomes progressively more challenging. However, the difficulty of track forecasting remains relatively consistent across these grades.\n    * 3. Impact of Data Volume:\n      * Models trained on all grades perform better in track forecasting compared to models trained on individual grades. This result indicates that a larger volume of training data improves track forecasting accuracy across different cyclone intensities.\n\n> Question #3: Apart from using negative indices, have you considered other mechanisms to separate visual and textual tokens (e.g., using token type embeddings)?\n\nThanks for the suggestion. \n1. We had explored an alternative mechanism, **offset indices**, where all image tokens are assigned an offset equal to the maximum length of the textual tokens. The results of this experiment are summarized in the table below:\n|Model|Intensity MAE|Intensity RMSE|Latitude MAE|Latitude RMSE|Longitude MAE|Longitude RMSE|Distance MAE|\n|---|---|---|---|---|---|---|---|\n|offset|1.815|3.762|0.188|0.303|0.229|0.385|30.157|\n|negative|1.515|2.981|0.084|0.159|0.095|0.178|16.275|\n\n    Based on this experiment, we found that our choice of using negative indices is an intuitive and efficient method for separating different types of tokens.\n\n2. Regarding using token type embeddings, we did not find evidence of their application in existing VLMs (e.g., Qwen [3], LLaVA [4], ViT [5], etc.). Moreover, the use of negative indices for image tokens inherently provides a natural and effective way to distinguish between visual and textual token types without additional complexity.\n\n\\\nFinally, we want to thank you once again for your thoughtful comments. We look forward to your feedback and we kindly ask that you consider these clarifications in your final grading.\n\nWarm regards,\n\nThe Authors\n\n\\\n[1] Kitamoto, Asanobu, Erwan Dzik, and Gaspar Faure. \"Machine Learning for the Digital Typhoon Dataset: Extensions to Multiple Basins and New Developments in Representations and Tasks.\" arXiv preprint arXiv:2411.16421 (2024).\n\n[2] Tarasiou, Michail, Riza Alp Güler, and Stefanos Zafeiriou. \"Context-self contrastive pretraining for crop type semantic segmentation.\" IEEE Transactions on Geoscience and Remote Sensing 60 (2022): 1-17.\n\n[3] Bai, Shuai, et al. \"Qwen2. 5-vl technical report.\" arXiv preprint arXiv:2502.13923 (2025).\n\n[4] Liu, Haotian, et al. \"Visual instruction tuning.\" Advances in neural information processing systems 36 (2023): 34892-34916.\n\n[5] Dosovitskiy, Alexey, et al. \"An image is worth 16x16 words: Transformers for image recognition at scale.\" arXiv preprint arXiv:2010.11929 (2020)."}, {"review_id": "ncqj7s8vhJ", "reviewer": "Reviewer_iSKv", "summary": "This paper proposes PIPE (Physics-Informed Position Encoding), a method that improves multimodal time series forecasting by embedding physical metadata—such as time, latitude, and longitude—into a vision-language model. PIPE consists of two key components: a physics-informed positional indexing scheme and a variant-frequency sinusoidal encoding that reflect real-world spatiotemporal structures. Applied to typhoon forecasting using satellite images and numerical data, PIPE outperforms existing models, achieving up to 12% improvement in intensity prediction.", "questions": "This reviewer would like to recommend that the authors please:\n\n1. Clarify how PIPE is novel compared to existing encoding methods like RoPE, relative position encodings, and 3D indexing.\n1. Provide evidence (e.g., ablations or probing) that the model meaningfully uses the injected physical encodings.\n1. Include experiments on PIPE’s generalizability to domains beyond typhoon forecasting.\n1. Compare PIPE to standard positional encoding baselines such as learned embeddings and RoPE.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths:**\n\n1. The paper clearly formulates a novel multimodal time series forecasting task that integrates satellite imagery with numerical sequences, addressing a realistic and impactful use case in climate forecasting.\n1.  PIPE is easy to implement and requires no additional learnable parameters, making it broadly applicable to existing vision-language models with minimal overhead.\n\n**Weaknesses:**\n1. While PIPE delivers promising empirical gains in multimodal time series forecasting, its core methodological design offers limited novelty. The physics-informed positional indexing maps known physical variables—such as timestamps, latitude, and longitude—to token position IDs using deterministic rules. This approach is structurally similar to absolute and 3D positional encoding schemes previously used in Transformers and Vision Transformers. Likewise, the variant-frequency sinusoidal encoding extends the standard Transformer positional encoding by introducing customized frequencies for each physical variable (e.g., hourly or geographic periodicity). While this results in a tailored encoding of spatiotemporal information, it remains an intuitive modification of the classic sinusoidal formulation.\n1. In addition, because these encodings are non-learnable and differ substantially from the token patterns observed during pretraining, it is unclear how effectively the underlying vision-language model can interpret them. Pretrained models like Qwen-2.5-VL are unlikely to have encountered structured positional patterns grounded in real-world physics (e.g., hourly timestamps mapped to unique negative IDs), and no mechanisms are introduced to ensure the model learns to associate these encodings with physical meaning. The integration thus relies entirely on the model’s ability to infer these semantics during fine-tuning, which may be unreliable. Without interpretability studies showing the model's reliance on the injected encodings, there is a risk that these handcrafted inputs are underutilized.\n1. The evaluation in this paper is limited to a single domain—typhoon forecasting using the Digital Typhoon dataset—which raises concerns about the generalizability of PIPE. Although the dataset is large and well-annotated, it represents only one type of spatiotemporal phenomenon under relatively consistent geographic and temporal conditions. It remains unclear whether PIPE can adapt to other multimodal forecasting tasks that differ in dynamics, data quality, or modality structure, such as precipitation estimation, air quality prediction, urban heat mapping, or traffic forecasting. These applications may involve different temporal resolutions, irregular sampling, heterogeneous sensor data, or weaker correlations between image and numerical modalities. \n1. The paper lacks a thorough comparison to alternative and widely-used positional encoding strategies, which limits the strength of its empirical claims. While ablation studies demonstrate the contribution of PIPE’s components (e.g., physics-informed indexing and variant-frequency encoding), the evaluation does not include baselines using other strong positional encoding methods, such as learned positional embeddings, relative positional encodings, or Rotary Positional Encoding (RoPE), which has proven effective in both language and vision tasks. These methods are well-established, often adaptable, and have demonstrated strong performance in capturing sequence order or spatial structure, including in long-context or multimodal settings. Moreover, since PIPE is non-learnable and fixed, while others are adaptive or trainable, it would be particularly valuable to compare performance, robustness, and interpretability across these approaches.", "was_revised": true, "final_justification": "While the paper formulates an interesting multimodal time series forecasting task and the proposed methods (PIPE) are easy to implement, the work as it presently stands has several weaknesses that my review mentions. Although authors’ rebuttal tried to address some of them, I remain unconvinced that there is a fundamentally new mechanism here. The proposed position encoding approach reuses well-established principles without introducing any fundamentally new mechanisms, and the authors rebuttal reinforces that assessment. The follow up response by the authors essentially reiterate the same points from the original rebuttal without providing new insights and thus does not not alleviate my fundamental concern.\n\nHowever, after looking at the exchange between the authors and the other reviewers as well as the clear message from the PC Chairs about seeking convergence, I have decided to suspend my concerns and have improved the rating from Borderline Reject to Borderline Accept, which is in line with other reviewers.", "rebuttal": "Dear Reviewer [iSKv],\n\nThank you for your time and comments.\n\n> Weakness #1: While PIPE delivers promising empirical gains in multimodal time series forecasting, its core methodological design offers limited novelty. ... it remains an intuitive modification of the classic sinusoidal formulation.\n>\n> Question #1:Clarify how PIPE is novel compared to existing encoding methods like RoPE, relative position encodings, and 3D indexing.\n\nThank you for acknowledging the empirical gains achieved by our method and for providing feedback on its novelty. The novelty of our method lies in its incorporation of the **global physical information** for vision tokens.\n\n1. **Comparison**: \n    * Existing positional encoding methods, including RoPE [1], relative PEs [2, 3] and 3D indexing [4], primarily focus on representing the absolute or relative order of tokens **within a single input sample**. These include absolute and relative positional encoding schemes, all of which operate at the level of individual samples and are not designed to encode global information. In contrast, our method captures **global physical information shared across all input samples**. It enables the model to better capture spatiotemporal relationships and leverage domain knowledge effectively among all input samples, as the physical knowledge remains unchanged.\n\n    * Additionally, we designed the encoding to encode **characteristics of specific physical variables**, allowing it to capture domain-specific periodicities (e.g., hourly or geographic cycles). This capability is limited or absent in existing positional encoding schemes and represents another key distinction of our approach.\n    * We have provided these clarifications in the **Introduction** (the paragraph prior to the contributions) and the **Related Work** section (‘Position Encoding in Transformers’). We appreciate the opportunity to clarify these points.\n\n2. **Novelty**: We respectfully argue that the value and novelty of a method should not be judged solely by whether it appears intuitive. Intuitive methods can be highly impactful, particularly when they result in substantial performance improvements, as demonstrated by our results. We believe that is the reason that Reviewer [8J4Q] and Reviewer [hPaZ] think we have a novel and elegant design of the scheme.\n\n> Weakness #2: In addition, because these encodings are non-learnable and differ substantially from the token patterns observed during pretraining, it is unclear how effectively the underlying vision-language model can interpret them. ... Without interpretability studies showing the model's reliance on the injected encodings, there is a risk that these handcrafted inputs are underutilized.\n>\n>  Question #2: Provide evidence (e.g., ablations or probing) that the model meaningfully uses the injected physical encodings.\n\nThank you for your feedback. We have used different ways to provide the evidence (including the ablations you suggested, and the attention visualization) to prove that our model meaningfully uses the physical informed encodings.\n\n1. Ablation Study:\nIn our ablation study, we systematically remove components one by one, such as 3D indexing, physics-informed indexing, and the variant-frequency sinusoidal function, to demonstrate they are meaningfully used. Reviewer [fqdc] also commented that our ablation study is comprehensive.\n\n2. Attention Visualization (Section E.5) for Interpretability:\nWe visualized the attention scores to further explain the model's behavior. Our analysis shows that with PIPE, the model's attention is concentrated on the typhoon region, whereas without PIPE, the attention becomes more diffuse. It demonstrates that the visual structure of typhoons are used for forecasting with PIPE.\n\nThese analyses provide evidence that the physical encodings meaningfully contribute to the model's performance.\n\n> Weakness #3: The evaluation in this paper is limited to a single domain ... These applications may involve different temporal resolutions, irregular sampling, heterogeneous sensor data, or weaker correlations between image and numerical modalities.\n>\n> Question #3: Include experiments on PIPE’s generalizability to domains beyond typhoon forecasting.\n\nThank you for your suggestion. To demonstrate the generalizability of our method, we added two additional experiments targeting different application domains: one focuses on typhoon forecasting in another region (the Australian (AU) region) [5]; the other uses the T31TFM-16 dataset for crop monitoring [6]. The first experiment confirms the robustness of our method in typhoon forecasting, while the second demonstrates its applicability to a different domain, showcasing its broader generalization capability.\n\n1. AU typhoon experiment:  \n  * Settings: The settings remain the same as PIPE. The focus is on typhoon forecasting, a regression task, which is the core of our paper.\n  * Result:\n|Models|Intensity MAE|Intensity RMSE|Latitude MAE|Latitude RMSE|Longitude MAE|Longitude RMSE|Distance MAE|\n|---|---|---|---|---|---|---|---|\n|Qwen-2.5-VL-3B (after tuning)|1.399|2.806|0.098|0.201|0.121|0.211|19.718|\n|PIPE-3B (Zero-shot)|1.361|2.773|0.094|0.171|0.118|0.206|19.090|\n|PIPE-3B (after tuning)|1.352|2.558|0.093|0.163|0.116|0.202|18.835|\n\n  * Analysis:\nThe results show that our method can be generalized to different regions for typhoon forecasting. Even the **model trained on the West Pacific region can perform well in the new dataset (zero-shot)**. For instance, compared to Qwen-2.5-VL-3B (after tuning), PIPE-3B (Zero-shot) achieved a lower intensity RMSE (2.773 vs. 2.806) and a smaller distance error (19.090 vs. 19.718). Additionally, PIPE-3B (after tuning) outperformed both models, achieving the best overall performance.\n\n2. Crop monitoring experiment:\n  * Setting: We adopt our method to the semantic segmentation task. The training data and test data were split by the original dataset, with a ratio of 0.85:0.15. We calculate latitude and longitude from the given UTM information provided in the dataset. We use the ViT model to encode the satellite image patches and incorporate each patch’s physical information. The focus is on crop monitoring, a segmentation task. Number of epochs: 50. lr: 1e-4. Optimizer: Adam.\n\n  * Result:\n|Models|IoU|Accuracy|\n|---|---|---|\n|ViT PE|67.52|80.61|\n|3D PE|67.27|80.43|\n|PIPE|69.21|81.81|\n\n  * Analysis: The results show that our method achieves the best result (69.21% and 81.81%) than standard ViT PE (67.52% and 80.61%)  and 3D PE (67.27% and 80.43%). The experiment shows that our method can be generalized to other domains by incorporating global physical information.\n\n> Weakness #4: The paper lacks a thorough comparison to alternative and widely-used positional encoding strategies, which limits the strength of its empirical claims.  ... it would be particularly valuable to compare performance, robustness, and interpretability across these approaches.\n>\n> Question #4: Compare PIPE to standard positional encoding baselines such as learned embeddings and RoPE.\n\nWe appreciate the reviewer's insightful feedback regarding positional encoding comparisons. \n1. We have compared our method with RoPE in the paper. In the main experiment (Table 1), ‘Qwen-2.5-VL-3B’ and in the ablation study (Table 2), ‘w/o 3D indexing (using sequence)’ are both using RoPE. We will add the clarification in the revised paper.\n\n2. Regarding the use of learned PE, we evaluated it prior to designing our main experiments. \n  * As noted in the literature [4, 7],  learnable PEs do not outperform deterministic ones. For instance, [7] states: “We also experimented with using learned positional embeddings instead and found that the two versions produced nearly identical results.”\n  * Another reason for not using learnable PEs in VLMs is that the **input sequence length can vary and may exceed** the maximum sequence length encountered during training. In such cases, learned PEs may fail to generalize. \n\n    These findings are reflected in modern architectures:\n    * RoPE: Used in Qwen [4] and LLaMA [8].\n    * Sinusoidal: Original Transformer [7] and other variants.\n\n    Therefore, we opted for deterministic PEs, as they offer the potential for the model to extrapolate to sequence lengths beyond those seen during training.\n\n\\\nFinally, we want to thank you once again for your thoughtful comments. We look forward to your feedback and we kindly ask that you consider these clarifications in your final grading.\n\nWarm regards,\n\nThe Authors\n\n\\\n[1] Su, Jianlin, et al. \"Roformer: Enhanced transformer with rotary position embedding.\" Neurocomputing 568 (2024): 127063.\n\n[2] Shaw, Peter, Jakob Uszkoreit, and Ashish Vaswani. \"Self-attention with relative position representations.\" arXiv preprint arXiv:1803.02155 (2018).\n\n[3] Ke, Guolin, Di He, and Tie-Yan Liu. \"Rethinking positional encoding in language pre-training.\" arXiv preprint arXiv:2006.15595 (2020).\n\n[4] Bai, Shuai, et al. \"Qwen2. 5-vl technical report.\" arXiv preprint arXiv:2502.13923 (2025).\n\n[5] Kitamoto, Asanobu, Erwan Dzik, and Gaspar Faure. \"Machine Learning for the Digital Typhoon Dataset: Extensions to Multiple Basins and New Developments in Representations and Tasks.\" arXiv preprint arXiv:2411.16421 (2024).\n\n[6] Tarasiou, Michail, Riza Alp Güler, and Stefanos Zafeiriou. \"Context-self contrastive pretraining for crop type semantic segmentation.\" IEEE Transactions on Geoscience and Remote Sensing 60 (2022): 1-17.\n\n[7] Vaswani, Ashish, et al. \"Attention is all you need.\" Advances in neural information processing systems 30 (2017).\n\n[8] Grattafiori, Aaron, et al. \"The llama 3 herd of models.\" arXiv preprint arXiv:2407.21783 (2024)."}, {"review_id": "F9Rr1NmJKG", "reviewer": "Reviewer_fqdc", "summary": "They propose a method to align different modalities which is physics informed, taking into account spatial and temporal information. Additionally, a separate positional encoding that uses variant-frequency is introduced for the physical variables. These should ideally help in typhoon intensity forecasting with the ability to integrate satellite imagery, which was not yet done before. The paper shows experiments that illustrate improvement over existing models, including models currently being used for current weather forecasting models. Ablations also show the contribution of each component of the proposed model.", "questions": "- why use Qwen-2.5-vl for the text and vision encoders?\n- what’s the motivation for freezing the vision encoder and not the text encoder?\n- on temporal position IDs: time has a “wrap around” property where the start of the next year is close to the end of the previous year. How is this taken into account (Eq 7)? Related, why are different components using different types of position IDs—why not just use the same type for all inputs?\n- Line 262-263: what is meant by the original dataset publication? is it the Digital typhoon dataset with the typhoon track annotations from Best Track?\n- Is the Qwen-2.5-VL baseline trained the same way as PIPE but without the multimodal alignment enhancements?\n- The PIPE contribution/improvement on the latitude and longitudes don’t seem big perhaps using UTM would better quantify the improvement (Tables 1,2)? 1 degree of lat/lon are quite large so the differences could be better emphasized in UTM since it’s in meters", "limitations": "The paper mostly talks about results having 12h lead time, is this generally enough to warn relevant agencies and for decision-making?\nThe distribution of the data was also not discussed which could pose limitations on the types of locations that this model can be applicable to.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n- Problem is well-motivated\n- Improvement with the addition of satellite image in forecasting\n- Prediction of typhoon positions seem accurate (Fig 3)\n- Ablation study shows contribution of each component of the proposed model\n- Extensive experiments that analyze the different aspects of PIPE such as showing decreasing errors with decreasing leading times, showing model improves with more introduced data (Appendix, Fig 5)\n\nWeaknesses\n- Design choices need to be further explained — i.e., instead of just saying “x was used…”, the rationale behind the choice could be further details (e.g., using Qwen as the text and vision encoders, freezing vision encoder, not freezing text encoder, using different positional indexing depending on the type of data such as sequential, 3D indexing, physics informed indexing, variant-frequency positional encoding)\n- Implementation details are not very clear, especially on the Qwen baseline and how it was trained, and how exactly it differs from PIPE —- is it just the position encoding that’s different?", "was_revised": true, "final_justification": "The paper introduces a new positional encoding method that takes into account the physics. I believe the paper supports this contribution sufficiently and shows comprehensive experiments and ablations to show the benefit of their method. The authors' rebuttal also addressed my concerns and most of the other reviewers' concerns, so I would like to maintain my positive review.", "rebuttal": "Dear reviewer [fqdc], \n\nThank you for your efforts and the positive feedback.\n\n> Weakness #1: Design choices need to be further explained — i.e., instead of just saying “x was used…”, the rationale behind the choice could be further details (e.g., using Qwen as the text and vision encoders, freezing vision encoder, not freezing text encoder, using different positional indexing depending on the type of data such as sequential, 3D indexing, physics informed indexing, variant-frequency positional encoding)\n>\n> Question #1: why use Qwen-2.5-vl for the text and vision encoders?\n>\n> Question #2: what’s the motivation for freezing the vision encoder and not the text encoder?\n\nThank you for your advice. We will include more detailed explanations of the selection of Qwen and freezing the vision encoder in the revised paper:\n1. Why use Qwen: \n    * This model incorporates 3D positional encoding (3D PE), which enhances its ability to align with video understanding. This feature is particularly relevant to our satellite image sequences that require such capabilities, making Qwen a suitable choice. \n    * Qwen demonstrates strong performance in multimodal alignment, which is crucial for our work, such as the MMMU benchmark [1]. \n    * It is an open-source model.\n2. Motivation for freezing the vision encoder and not the text encoder:\n    * We need to fine-tune the LLM during the SFT stage to equip it with domain-specific knowledge, ensuring it becomes trainable for our tasks. \n    * For the vision encoder, we observed that the original encoder is already effective in capturing visual patterns, and additional training did not yield further benefits. The experimental results are as follows:\n|Model|Intensity MAE|Intensity RMSE|Latitude MAE|Latitude RMSE|Longitude MAE|Longitude RMSE|Distance MAE|\n|---|---|---|---|---|---|---|---|\n|unfreezing the vision encoder|1.633|3.307|0.084|0.155|0.101|0.184|16.696|\n|PIPE (freezing)|1.515|2.981|0.084|0.159|0.095|0.178|16.275|\n\n3. Regarding the selection of different components, we have conducted a comprehensive ablation study to validate their effectiveness, as highlighted in your **Strength #4**. The results demonstrate that each selection of the components is reasonable.\n\n> Weakness #1: Implementation details are not very clear, especially on the Qwen baseline and how it was trained, and how exactly it differs from PIPE —- is it just the position encoding that’s different?\n>\n> Question #5: Is the Qwen-2.5-VL baseline trained the same way as PIPE but without the multimodal alignment enhancements?\n\nThank you for your feedback. Yes, Qwen and PIPE differ only in their positional encoding to evaluate the effects of our methods. Both were implemented using LLama-Factory on NVIDIA H800 GPUs, optimized with AdamW, an initial learning rate of 1e-5, a batch size of 1, and cross-entropy loss over one training epoch. **The code has been included in our submission.**\n\nThese details are also outlined in Section 4.2 (line 264) and Appendix C: Implementation Details.\n\n> Question #3: on temporal position IDs: time has a “wrap around” property where the start of the next year is close to the end of the previous year. How is this taken into account (Eq 7)? Related, why are different components using different types of position IDs—why not just use the same type for all inputs?\n\nThank you for your question. \n1. In the Eq7, t is the hourly progression of the year.  During training, the attention mechanism can capture the relationship between the end of one year and the start of the next if there are \"wrap-around\" patterns present in the training dataset.\n2. We used different types of positional IDs for textual tokens and vision tokens. Vision tokens encode satellite image patches that contain physical information such as time, latitude, and longitude, while textual tokens only encode their sequential order.\n\n> Question #4: Line 262-263: what is meant by the original dataset publication? is it the Digital typhoon dataset with the typhoon track annotations from Best Track?\n\nThank you for your question.\n1. The \"original dataset publication\" refers to the open-sourced dataset cited in [2]. \n2. Yes, the annotations in this dataset are derived from the Best Track dataset, as described in Section 4.1 (line 244).\n\n> Question #5: The PIPE contribution/improvement on the latitude and longitudes don’t seem big perhaps using UTM would better quantify the improvement (Tables 1,2)? 1 degree of lat/lon are quite large so the differences could be better emphasized in UTM since it’s in meters\n\nThank you for your suggestion. \n1. Using longitude and latitude is a standard practice in typhoon research, especially given the large scale of the study (as illustrated in Figure 1). \n2. Moreover, the data were normalized in the model, which allows the **differences to be effectively emphasized**.\n3. We agree that converting coordinates to UTM may offer further advantages. At this stage, meanwhile, we believe that the choice of coordinate transformation does not materially affect the core contribution of our work, as our focus is on validating the effectiveness of the physics-informed position encoding framework itself. We will exploring alternative coordinate systems, include UTM, in our future work.\n\n> Limitation #1: The paper mostly talks about results having 12h lead time, is this generally enough to warn relevant agencies and for decision-making? The distribution of the data was also not discussed which could pose limitations on the types of locations that this model can be applicable to.\n>\n> Ethical Concerns: Data quality and representativeness\n\nThank you for your question.\n1. **Short-term forecasting is of critical importance** for timely decision-making, as typhoon events typically span only a few days. For example, the maximum forecasting length is limited to 12 hours in the dataset baseline [2] used in our study and limited to 1 hour in [3], which aligns with the practical need for short-term predictions. \n\n    We have also included the extended forecasting horizons for VLMs in the Limitations section (line 642). Addressing the challenge of long-term forecasting will be one of the key focuses of future work, particularly addressing the limitation of current VLMs to handle more input images. However, the **primary goal of this paper is to explore whether PIPE is effective** in incorporating additional physical information into forecasting models. We believe that our experimental results successfully demonstrate the validity of our methods.\n2. The quality of the dataset: The dataset [2] we used is of high quality, as it was recognized with a Spotlight at NIPS. Additionally, the dataset is representative, encompassing all historical typhoons from 1978 to 2023.\n\n\\\nFinally, we want to thank you once again for your thoughtful comments. We look forward to your feedback and we kindly ask that you consider these clarifications in your final grading.\n\nWarm regards,\n\nThe Authors\n\n\\\n[1] Yue, Xiang, et al. \"Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.\n\n[2] Kitamoto, Asanobu, et al. \"Digital typhoon: Long-term satellite image dataset for the spatio-temporal modeling of tropical cyclones.\" Advances in Neural Information Processing Systems 36 (2023): 40623-40636.\n\n[3] Zhu, Jiakai, and Jianhua Dai. \"A rain-type adaptive optical flow method and its application in tropical cyclone rainfall nowcasting.\" Frontiers of Earth Science 16.2 (2022): 248-264."}]

Author rebuttal:

Meta-review: **Paper Summary:**\
This paper proposes a novel multimodal time-series forecasting framework called PIPE that incorporates visual information through a vision-language model. The paper presents extensive experiments and has been broadly recognized for its innovation. During the rebuttal phase, the authors engaged in in-depth discussions with the reviewers, leading some reviewers to improve their ratings.

**Justification:**\
Given that the reviewers unanimously recommended acceptance, and the authors engaged in multiple rounds of in-depth discussions that adequately addressed the reviewers’ concerns, I recommend accepting the paper.

**Summary of Rebuttal Period:**\
Reviewers NJNg, 8J4Q, hPaZ, and iSKv appreciated the paper’s innovation and applicability, while Reviewers NJNg and fqdc considered the experiments adequate. Reviewer 8J4Q noted that the paper was well-organized. During the rebuttal phase, the authors engaged in in-depth discussions with the reviewers and successfully addressed most of the concerns, which led to some reviewers raising their scores.
Decision: Accept (poster)