Eyes on the Road, Words in the Changing Skies: Vision-Language Assistance for Autonomous Driving in Transitional Weather
Authors: Madhavi Kondapally
K Naveen Kumar
C Krishna Mohan
Venue: Accepted by TMLR ()

Abstract: The rapid advancement of autonomous vehicle technology (AVT) necessitates robust scene perception and interactive decision-making, particularly under adverse weather conditions. While significant progress has been made in extreme weather scenarios like cloudy, foggy, rainy, and snowy, a critical challenge remains in transitional weather conditions, such as the shift from cloudy to rainy, foggy to sunny, etc. These dynamic environmental changes degrade the performance of conventional vision-language systems by causing unpredictable illumination changes and partial occlusions, which are inadequately represented in current AVT datasets. This lack of continuous, transitional training data compromises model robustness and ultimately affects safety and reliability. On the other hand, Vision-language Models (VLMs) enable interpretable reasoning in autonomous driving through tasks such as image captioning and visual question answering. However, current VLMs, designed for clear weather, perform poorly in transitional conditions and rely on computationally expensive LLMs. This leads to high memory usage and slow inference, which is unsuitable for real-time decision making in AVT. To address these limitations, we propose Vision-language Assistance for Autonomous Driving under Transitional Weather (VLAAD-TW), a lightweight framework with a novel cross-modal spatiotemporal reasoning architecture that robustly interprets and acts on multimodal data. The VLAAD-TW framework integrates a Feature Encoder for Transitional Weather (FETW), a lightweight backbone for robust visual feature extraction, with a Spatiotemporal Contextual Aggregator (SCA), which models dynamic weather-induced changes. It uses a Selective Attention-guided Fusion Module (SAFM) to balance visual and linguistic cues for a unified representation dynamically. Finally, a Semantic Text Generator (STG) fuses these representations to produce context-aware driving information, adapting in real time to both current and predicted weather states. Further, we introduce the AIWD16-text dataset, an adverse intermediate weather driving dataset for vision language tasks, which features sixteen transitional weather states created using a Stochastic Conditional Variational Autoencoder (SC-VAE) and enriched with manual annotations of image captions and open-ended question-answer pairs. An extensive evaluation of the AIWD16-text and DriveLM datasets demonstrates VLAAD-TW's high performance in BLEU and ROUGE scores, with low memory and computational requirements, confirming its effectiveness in challenging weather conditions.

Published in Transactions on Machine Learning Research (01/2026)
Eyes on the Road, Words in the Changing Skies: Vision- Language Assistance for Autonomous Driving in Transitional Weather
Kondapally Madhavi
cs21resch15001 wwth:50000@@t...ooo ac.in
Department of Computer Science and Engineering Indian Institute of Technology Hyderabad
K Naveen Kumar
cavenbbumorr.csee@aaamma ac.in
Department of Computer Science and Engineering Indian Institute of Technology Hyderubad
C Krishna Mohan
ckm commdcce.itth.cm
Department of Computer Science and Engineering Indian Institute of Technology Hyderabod
Reviewed on OpenReview: https: opt.r//eepereeeeeeeem net/ forum? id= PCEDudVJon
Abstract
The rapid advancement of autonomous vehicle technology (AVT) necessitates robust scene perception and interactive decision-making, particularly under adverse weather conditions. While significant progress has been made in extreme weather scenarios like cloudy, foggy, rainy, and snowy, a critical challenge remains in transitional weather conditions, such as the shift from cloudy to rainy, foggy to sunny, etc. These dynamic environmental changes degrade the performance of conventional vision-language systems by causing unpredictable illumination changes and partial occlusions, which are inadequately represented in current AVT datasets. This lack of continuous, transitional training data compromises model ro- bustness and ultimately affects safety and reliability. On the other hand, Vision-language Models (VLMs) enable interpretable reasoning in autonomous driving through tasks such as image captioning and visual question answering. However, current VLMs, designed for clear weather, perform poorly in transitional conditions and rely on computationally expensive LLMs. This leads to high memory usage and slow inference, which is unsuitable for real-time decision making in AVT. To address these limitations, we propose Vision-language Assis- tance for Autonomous Driving under Transitional Weather (VLAAD-TW), lightweight framework with novel cross-modal spatiotemporal reasoning architecture that robustly interprets and acts on multimodal data. The VLAAD-TW framework integrates a Feature Encoder for Transit ional Weat her (FETW), a lightweight backbone for robust visual feature extraction, with a Spatiotemporal Contextual Aggregator (SCA), which models dynamic weather-induced changes. uses a Selective Attention-guided Fusion Module (SAFM) to balance visual and linguistic cues for unified representation dynamically. Finally, a Se- mantic Text Generator (STG) fuses these representations to produce context-aware driving information, adapting in real time to both current and predicted weather states. Further, we introduce AIWD16-text dataset, an adverse intermediate weather driving datasel. for vision language tasks, which features sixteen transitional weather states created using Stochastic Conditional Variational Autoencoder (SC-VAE) and enrichod with manual anno- lations image captions and open-ended question-answer pairs. An exlensive evaluation of the ATWD16-text and DriveLM datasets demonstrates VL'ADD-TW's high performance in BLEU and ROUGE scores, with low memory and computational requirements, confirming its effectiveness in challenging weather conditions.

---

Published in Transactions on Machine Learning Research (01/2026)
1 Introduction
Autonomous vehicle technology (AVT) has made substantial progress in advanced driver assistance systems for real-world applications (Li et al., 2024b] Hwang et al., 2025). However, the functionality of AVT is significantly challenged by adverse weather conditions, such as rain, fog, and snow (Wang et al.] 2024 Lu ct al.) 2025). Also, these weather scenarios drastically degrade the quality of visual input from camera sensors, posing difffculties for vision-language models (VLM) lo generate accurale and reliable descriptions of driving scenes (Xie et al.) 2024 Luo et al.) (025).. Moreover, sudden weather changes, referred to as transitional weather conditions like cloudy to rainy rainy to foggy, obstruct the recognition of vehicles, pedestrians, and traffic signs due to variations in weather intensity (Sun et al. 2022
Need for the study of transitional weather and its dataset. Transitional weather refers to interme- diate weather conditions that occur when weather changes from one type to another. It involves gradual shifts in weather conditions. Examples include transitions such as foggy weather turning into rainy or rainy shifting to cloudy, with variations the weather intensity (Sun et al.] 2022). Studying transitional weather conditions is critical in real-world environments, as abrupt transitions between weather conditions are com-
mon and pose challenges to AVT. Existing datasets (Caesar et al., 2019 Sakaridis et al. 2018) primarily focus on either clear weather extreme conditions, leaving a notable gap in understanding and modeling transitional weather scenarios. To address this limitation, we introduce the AIWD16-text dataset, designed for vision-language assistance in AVT. The dataset encompasses sixteen transitional weather scenarios, in- cluding cloudy to rainy (CR), sunny to foggy (SF), sunny to rainy (SR), cloudy to snowy (CSn), cloudy to foggy (CF), snowy to rainy (SnR), snowy to foggy (SnF), foggy to rainy (FR) and vice versa. Additionally, we provide manually annotated, generated images to facilitate the creation of open-ended question ansserr pairs, enabling richer vision-language modeling under challenging weather transitions. To our knowledge, this is the first dataset tailored for image captioning and visual question-answerinn (VQA) tasks for AVT under transitional weather conditions.
Transitional weather impact. on VLMs. Vision-language models (VLMs) offer a promising avenue for interpretable reasoning in autonomous driving tasks to interpret their surroundings semantically (Ma et al, 2024b Moeller et al., 2025). They enable tasks like image captioning (e.g.. "A pedestrian is crossing the street') and visual question answering (e.g., 'Is the traffic light red?"), which crucial for enhancing situa- tional awareness and providing interpretable basis for real-time decision-making in autonomous driving systems (Kuchibhotla et al. 2025 Zhang et al., 2024a). However, existing VLMs (Chen et al. 2024b) are typically designed for static, clear weather and often fail to perform reliably in dynamic transitional weather scenarios due to high visual variability caused by changing weather patterns and reduced illumination, mak- ing it challenging to detect and identify crucial road elements, such as pedestrians and vehicles. Moreover, a key barrier to their adoption in AVT their reliance on computationally expensive Large Language Models (LLMs), which leads high memory demands and slow inference times, rendering them unsuitable for the real-time decision-making required for autonomous driving (Gopalkrishnan et al. 2024).
Limitations of existing methods. Table 1 presents summary of existing VLMs, highlighting the focus of our work on transitional weather conditions. While current VLM research on image captioning and VQA (Li
et al. 2022 Byun et al., 2024 Li et al. 2023) focuses predominantly on general-purpose applications, these models lack exposure to driving-specific scenarios, limiting their ability to understand crucial road elements and safety-critical situations. Similarly, existing VQA methods for AVT are primarily limited to ideal daylight conditions, failing to address real-world complexities such as transitional weather conditions (Gopalkrishnan et al., 2024 Park et al., 2024). The limited availability of training data representing these diverse weather scenarios further compounds the performance constraints. Although few VLMs target specialized tasks like traffic rule extraction (Li et al.] 2024a) and action explanation (Ma et al. datasets and controlled environments constrains their elfectiveness in
2024b), their reliance on curated handling unpredictable real-world 2025). Moreover, these approaches
driving conditions (Kuchibhotla et al., 2025] Wu et al. 2025 Zang et al. typically depend on large-scale models (Li et al.) 2022; Touvron et al.) with over a billion paramcters, requiring expensive hardware and suffering from slower inference speeds, making them impractical for real- time autonomous driving. Overall, existing methods suffer from three key limitations, (i) insufficient datasets
2

---

Published in Transactions on Machine Learning Research (01/2026)
Table 1: Comparison existing vision language methods with the proposed method. R-+Rainy, C-Cloudy, S S-Sunny, F Foggogy, Sn Sn-tSnowy, Present,, X-NNot present, ' Noo applicable.
Transition,
Autonomous driving scenarios
Adverse weather conditions
Data generation
transition classes, and real data
Method
Task
CLIP RRdford al.
2021
Conoept-gridlook
Echternot
2024
k.
IL-CLIP Zheng et a.
2121
X
X
x
MAFA
Bynn Dt. al.
Image Captioning
VeCLIP
Lai
x.
EXCLIP
Moeller e5
2722
X
x
SOLO
Then et SJ.
21
21b
X
x
x
LiugoQA QA
Miaren el ml. 21124
EM. VLM4AD Gopalkrishi
2024
C. R.
VLAAD
[Park
2024
VQA
x
CocoCoon
Mahtaaan
al. (24224
X
x
x
CODA-VLM
( hen et al
2025
/
L.aDA
ml.
Traffic rule extraction
x
LaMPilos
x.
Driving actions and explutations
SCDCL
[Cao et
20221
/
x
BLT
LL et a
2022
x
Image Captioning VQA
BLIP-2 of al:
2023
V-AAD-TW [Ours]
R. Sn
16.
for transitional weather conditions, (ti) limited output interpretability, and (tti) deployment constraints of large-scale models real-time systems.
To address the above challenges, we propose VLAAD-TW, a lightweight vision-language assistance framework specifically designed for autonomous driving under transitional weather conditions. The overview of the VLAAD-TW framework, illustrated in Figure I is designed for image captioning and VQA. VLAAD-TW introduces novel cross-modal spatiotemporal reasoning architecture that robustly interprets and acts on multimodal data in transitional weather conditions. The framework integrates (a) a Feature Encoder for Transitional Weather (FETW), a lightweight residual backbone fine- tuned to maintain feature stability across varying intensities of clouds, rain, fog, and snow, (b) a Spatiotemporal Contextual Aggregator (SCA), that jointly captures spatial scene structure and temporal evolution to model dynamic ienther-induood changes, (c) Selective Attention-guided Fusion Module (SAFM) that dynamically balances visual cues (e.g. obstacles) and linguistic descriptions (e.g., 'heavy rain,' 'low visibility") to form a unified safety-critical reprosentation, and (d) a Semantic Text Generator (STG) that fuscs these representations to produce context- aware driving information, adapting in real time transitional weather states. Additionally, we introduce AIWD16-text, a novel vision language dataset tailored for transitional weather scenarios, created using a stochastic conditional variational autoencoder (SC-VAE) to enhance model diversity and generalization in adverse and transitional conditions. This dataset is manually annotated support tasks such as image captioning and VQA, offering valuable resource for advancing research in this domain. To the best of our knowledge, this work represents the first comprehensive effort address vision-language tasks in transitional weather conditions for AVT. The main contributions of our work are as follows.
1. A novel, end-to-end framework for Vision-Language Assistance in Autonomous Driving (VLAAD-TW). We introduce first-of-is-kinnd model specifically developed to perform complex VQA and image captioning tasks under challenging transitional weather conditions, addressing a critical and previously underexplored challenge in aulonomous vehicle perception.
2. A lightweight and robust architectural design for transitional weather perception. Our VLAAD-TW model incorporates specialized Feature Encoder for Transitional Weather (FETW) and Spatiotemporal Contextual Aggregator (SCA). This design ensures both computational efficiency for real-time applications and robust feature stability, mitigating the adverse effects of transitional weather phenomena like cloudy to rainy and rainy to foggy, etc., on perception.
3. AIWD16-text dataset. We introduce AIWD16-text, a novel rision-languse dataset generated using a stochastic conditional variational autoencoder (SC-VAE) to address the challenges of transitional weather
3

---

Published in Transactions on Machine Learning Research (01/2026)
Sunny to Foggy
Sunny t=1
Foggy
t= 2
t=T
Query: What is the current weather scenario and describe the current scene?
Response:
The current weather is moderate fog with cars moving on the road and few bikes parked right side.
VLAAD-TW
Feature Encoder (FETW)
Context Aggregator (SCA)
Attention-guided Module (SAFM)
Semantic Text Generator (STG)
Figure 1: Overview of the proposed VLAAD-TW approach: Transitional weather images generated by the SC-VAE are input to VLAAD-TW, which generates captions and answers to provide vision-language assistance.
conditions autonomous driving. This datasct, which includes a total sixteen distinct weather states, features a rich combination of manually curated image captions and open-ended visual question-answer pairs, making it an ideal resource for training and evaluating languae models in this domain.
4. Comprehensive evaluation. We present a comprehensive evaluation of our VLAAD--W framework, benchmarking its performance against several state-of-the-art vision-language models. This evaluation is conducted both our new AIWD16-text dataset and the well-established DriveLM dataset. Furthermore, provide detailed analysis of VLAAD-TW's computational complexity, demonstrating its efficiency and roal-world applicability compared to existing, larger state-of-thc-art models.
Section 2 reviews prior work on synthetic weather generation, vision-language models (VLMs) for image captioning and visual question answering (VQA), and the application of VLMs in AVT. Section 3 discusses essential background information for weather transition generation. Section ! iillustrates the details of our proposed method, which comprises two stages, transitional weather data generation and vision language modeling for autonomouu driving. Section 5 presents a comprehensive assessment of our proposed method for dataset generation and vision language modeling tasks. Section ablation study of work. Section pection[]provideee provides limitations of our work and future research directions. Finally, Scction 8 summarizes key research findings.
Paper outline:
0 provides a detailed
2
Related Work
This section discusses recent works on vision language models for image captioning, visual question and answer (VQA) and other notable works.
4

---

Published in Transactions on Machine Learning Research (01/2026)
2.1 Synthetic Weather Generation
To address the challenges of limited data availability for AVT in inclement weather, numerous studies (Diaz
et al.) 2022 Li ct al., 2025 Marathe et al. 2023) have presented methodologies for generating synthetic
data in autonomous vehicles. Park et al. (2024) introduced multi-modal instruction tuning dataset to
facilitate language models in learning instructions across driving scenarios. Marcu et. al. (2024) introduced the LingoQA benchmark for visual question answering in autonomous driving. Li et al. (2025) proposed a spatial-temporal consistent diffusion framework, DrivingDiftusion, to generate realistic multi-viow videos for autonomous driving. MMrathe et al. (2023) introduced WEDGE, a synthetic dataset generated with a vision-language generative model via prompting. El-Shair et al. (2024) introduced a large-scale stereo image dataset that captures a wide spectrum of challenging real-world environmental scenarios. Muşat al. (2021)
generated seven diverse weather conditions using GAN and CycleGAN. Hu et al. (2021b) synthesized rainy images to create a new rainy dataset called rainy cityscapes for single-image rain removal. Most of these methods generate discrete weather conditions. Unlike these, [Snn et al. (2022) introduced a synthetic driving dataset known as SHIFT for performing scene perception tasks under discrete and contimuous domain shifts using a Carla simulator (Dosovitskiy et al.) 2017). However, compared to generative-based approaches, simulators often fall short variability, realism, complexity, and scalability. To overcome these limitations, we generate the transitional weather dala using generalive approaches that allow us to vary wealher intensilies and annotate them to create rich set of open-enden question -answer pairs for driving scenarios.
2.2 VLMs for Image Captioning and VQA
Vision-language models connect. the capabilities of language and vision models by integrating images and text into shared latent representation. They utilize cross-modal pre-training tasks to establish relationships between visual and textual data, inspiring the creation of numerous models in multimodal learning (Gopalkr- ishnan et al., 2024 Li et al. (022) proposed BLIP, unified framework for vision-language understanding and generation by bootstrapping pre-training with noisy web data. Li et al. (2023) proposed BLIP--2 that improves upon BLIP by utilizing frozen image encoders and large language models for vision- language pre- training. Zheng et al. (2024) design an iterated learning algorithm that improves compositionality in large vision-language models. Radford et al. (2021) proposed a pre-training task that pairs text captions with images, resulting in CLIP, which learns advanced image representations and demonstrates exceptional zero- shot transfer performance across various image classification tasks. Byunn et al. (2024) introduced MAFA, specifically designed to tackle false negatives in VLP. Ma et al. (2024a) introduced vision-language in- struction tuning (VLIT) approach using contrastive learning (C3L), featuring a content relevance module to align VLIT data with images and a contrastive lcarning module to enhance data generation. Shukor eet al. 2022) proposed ViCHA, incorporating three key components: hierarchical cross-modal rast alignment, self-supervised masked image modeling, and supervision using cerieeerived Visual Concepts. Li & JJian (2024) proposed method that leverages cross-modal pre-training and semantic modcling to en-
hance video anomaly detection, achieving improved interpretability and performance. Moeller et al. (2025) introduced second-order attribution method for dual-encoder models, showing that when applied to CL.IP.
it effectively reveals fine-graines correspondences between caption tokens and image regions. Although the proposed methods are efficient and focus predominantly on general-purpose applications, these models lack exposure to driving-speciftc scenarios, which limils their ability to undersland critical road elements and safety-related situations.
2.3 VLMs for Autonomous Driving
Autonomous driving systems predominantly rely visual features, but incorporating linguistic features can improve their interpretabilityy and aid in identifying novel traffic scenarios. This advantage has fueled grow- ing research into leveraging multimodal data to train language models as autonomous driving agents (Ma) et al. 2024b). Gopalkrishhan et al. (2024) proposed EM-VLMAAD, a multi-frame vision-language model optimized for efficient visual question answering in autonomous driving, focusing on memory efficiency and reduced computational demands. |ark et al. (2024) introduced VLAAD, a multimodal LLM driving assis-
tant designed to generate detailed captions for autonomous driving scenarios. Pan et al. (0024 presented
5

---

Published in Transactions on Machine Learning Research (01/2026)
Table 2: Summary of adopted notations
Notation
Definition
D
Training set
M
Size of training set
FG. yi
Input image pair (Ex. 2;: Sunny image and yi: Rainy image) One-hot encoded labeled vector
4
Variational parameters
Latent variable
Input features
Model parameters
μ
Mean gaussian distribution
Variance of gaussian distribution
3
Coefficient
Loss
Transition sequence
T
Length of transition sequence Coefficients
8o & S1
Error
à
Reconstructed input Covariance
V
Feature vectors obtained from CNN
Processed feature vectors
Attention vector
C
Combined context vector
h
Hidden state
vision-language planning framework to bridge the gap between linguistic understanding and autonomous driving. Chen et al. (2024)) developed architecture that combincs vectorized numeric modalities with pre-trained LLaMA-7b Touvron al. to tackle driving question-answwriin tasks. Their two-step training ap- proach involves grounding vector representations into interpretable embeddings for the frozen LLaMA model, then fine-tuning the LLM using LoRA (Hu et al. 2021a). Similarly, DriveGPT4 Xu et al. (2024) utilizes LLaMA as its backbone language model and CLIP as a visual encoder, processing traffic seene videos and prompt. text to generale responses and low-level vehicle rontrol signals. In contrast, Li LLaDA, vision-language framework capable of interpreting traffic rules in new locations with zero-shot gen- eralizability. Ma et al. (2024b) introduced framework integrating LLMs into autonomous driving systems, enhancing their ability to interpret and follow commands. Additionally, Cao al. (0244 proposed the Comantic-Guided Dynamic Correlation Learning (SGDCI.) framework, leveraging semantic-guided and dynamic correlation modules to enhance explainable autonomous driving. However, these methods remain largely limited to ideal daylight conditions, failing to address real-world complexities such as transitional weather. Hence, in our work, we propose a lightweight and efficient vision iision-language assistant capable of generating captions and responding queries autonomous driving scenarios under transitional weather conditions.
2024a) proposed
3 Preliminaries
This section provides the essential background information required to comprehend the remainder of the paper. A summary of the notations used throughout this work is presented in Table 2
3.1 Latent Space Representation
The latent space of a Conditional Variational Autoencoder (C-VAE) a hidden, low Now-dimessional repre- sentation of the input data. Unlike a traditional autocncoder, which maps an input a single point, the C-VAE learns probabilistic distribution, specifically, Gaussian for each input. This approach prevents overfitting and ensures that the latent space is continuous and well-structured, property that is essential

---

Published in Transactions on Machine Learning Research (01/2026)
for a generative model. The inherent smoothness of this latent space is key feature of the C-VAE, it enables seamless and meaningful interpolation between distinct data points. By generating a new point in the latent space and decoding it, the model can synthesize novel data that blends the characteristics of the original inputs. This property is crucial for our work, allowing us to generate the progressive weather transitions required for our dataset.
3.2 Data Interpolation
To generate a diverse and continuous spectrum of transitional weather states for the proposed AIWD16-text dataset, we leverage the power of latent space interpolation within C-VAF. The process begins by encoding two distinct input images captured under different weather conditions to obtain their corresponding latent vectors, 2z and Zy, and their one-hot encoded condition vectors, 1e and 1V, A series of intermediate latent vectors, 2, and blended condition vectors, I, are then generated via linear interpolation, The C-VAE's decoder then utilizes these interpolated latent and condition vectors to transform them into a progressive sequence of images. This technique allows for synthesising a smooth and realistic evolution of visual features wetween distinct weather conditions, with 3 providing fine-grained control over the blend of characteristics.
3.3 Transitional Weather Conditions
Weather transitions are common in the real world and pose a challenge for AVT, particularly in rain, clouds, and fog. Existing datasets typically capture stable weather conditions (e.g., cloudy or rainy), but real-world transitions, such as from sumny to rainy, involve varying intensities. For example, transitioning from sunny to rainy involves intensities ranging from light to heavy rain (Sun et al., 2022). Recent studies highlight challenges posed by continuous weather shifts for learning systems, underscoring the need for datasets that capture dynamic weather changes (Sun et al. 2022 Our work provides such a dataset using customized conditional variational autoencoder, enabling vision language modeling for AVT development under transitional weather conditions.
Importance of addressing weather transitions in autonomous vehicles. Transitional weather condi- tions signilicantly impact scene perception performance in aulonomous driving. Although existing dalasets such as Foggy Cityscapes (Sakaridis et al. 2018) and MultiWeatherCity (Musat et al. 2021) capture ad- verse weather sccnarios, they mainly focus on discrete weather conditions such as heavy rain or intense fog. On the other hand, real-world driving environments often involve gradual weather transitions (for example, foggy to rainy or rainy to sunny). These transitions introduce additional complexities that go beyond those encountered in extreme weather conditions.
Unique challenges posed by transitional weather. Existing AVT datasets and models inadequately address critical challenges associated with weather transitions: (i) Unlike static weather states, transitions involve dynamic changes in visibility and illumination, which can disrupt perception models trained on fixed conditions, (ii) autonomous systems rely on stable environmental features for perception. However, weather transitions introduce inconsistencies in visual cues, leading to misclassification or unreliable predictions.
Limitations of existing models in handling transitional weather. Most VLMs are trained on datasets with extreme or static weather conditions, making them ill-equipped to handle gradual transitions. Their primary limitations include: (i) Models trained clear extreme weather conditions lack the adaptability to generalize across intermediate weather changes, (ii) existing models often rely on static images or short sequences, missing the gradual evolution of visual ures over time, (iii) during transitions, environmental features may not match any single weather condition category, causing prediction errors. These limitations roduce model robustness real-world deployments, where smooth weather transitions are common. Ad- dressing these gaps essential for developing more reliable and adaptable autonomous vehicle perception systems.
7

---

Published in Transactions on Machine Learning Research (01/2026)
Sunny
Stochastic Conditional Variational Autoencoder
(SC-VAE)
Foggy
Encoder
Docoder
Sunny to Foggy
t=1
2
t= T
Figure 2: The proposed transitional weather data generation approach using customized conditional vari- ational autocncoder. Here [0. 0] for sunny weather and 1, for foggy weather.
4 Proposed Method
Method overview. Our proposed method comprises two stages: (i) generate benchmark transitional weather dataset for AVT and (ii) perform vision-language tasks such as image captioning and VQA on the generated dataset. Figure 2 presents our proposed transitional weather data generation approach. First, we input driving scene images x and a one-hot encoded vector representing the corresponding weather condition into the SC-VAE. The SC-VAE's encoder processes both the image features and the condition vector to generate the parameters (μ, log o") of a latent distribution. To synthesize transitional images, perform linear interpolation between two latent embeddings while using blended condition vector to guide the decoder. This process allows the docoder to produce continuous and controlled evolution of visual features, resulting in a sequence of transitional weather images with varying intensities Annolations are then manually created to support both vision- language tasks effectively. Figure 3 illustrates the framework of our proposed VLAAD-TW architecture, which intogrates a scries of components to effectively generate textual descriptions and answers to questions from images. A sequence of transitional weather images first input to the Feature Encoder for Transitional Weather (FETW), a lightweight MobileNet backbone (Sandler al., 2018) that extracts high-level visual features resilient to weather-related degradation. Concurrently, input text is processed by the Spatiotemporal Contextual Aggregator (SCA), which consists of series of LSTM networks that transform the word sequence into coherent, context-aware representation. The outputs from both the FETW and SCA modules are then passed to the Selective Attention-Guided Fusion Module (SAFM). This module dynamically weighs the importance of both visual features and the linguistic context, fusing them into a unified, salient representation. Finally, the fused context is fed to the Semantic Text Generator (STG), which consists of LSTM layers that combine the context vector with previous outputs maintain sequence coherence. A dense layer with softmax activation predicts the next word, generating a refined and context-aware lextual output.
4.1 Transitional Weather Data Generation
Let D = [(x,, Wi, H4, ( denole the training dataset, where each pair of images (xi, w: is captured under two distinct stable weather conditions. The corresponding one-hot encoded labels for these conditions are denoted by i and 14, respectively, and M the total number of training samples. Our stochastic conditional variational autoencoder (SC-VAE) consists of an encoder, denoted as qu (2 xx 1), which processes each image and its condition vector independently. For given input pair (I), the encoder is applied to each image
8

---

Published in Transactions on Machine Learning Research (01/2020)
Processed feature vector
Ilideen states
Combined vector Sequence output
Output
Feuture propagation
Fenture
embeddings
Input
Current weather: Sanny
Sunay
Feature Encoder (FETW)
Somantic Text Generator (STG)
pedestrians & cars moving on read
PL
LSTM
LSTM
Danee layens
LSTM
LSTM
weather:
Light Fog
Dones layons
pedestrians & cars moving
Reppor
Switmax
Question and unawer Unpot tokens)
Spatiotemporal Context Aggregator (SCA)
west sooel secee
ESTM
M
Emiroliing loyer
Current weather: Heavy Pog pedestrians & cars moving read
LSTM
Question: W What is the weather secuario descelhee the current scune?
ESTM
LSTM
Attention-guided (SAFM)
Figure The proposed architecture of the VLAAD-TW. We imput transitional weather sequences (e.g. Sunny to Foggy) the Feature encoder (FETW) having MobileNet backbone for feature extraction, re- fined by dense layers, and contextualized using context generator (SCA) and attention-guided fusion module. The semantic text generator (STG) integrates the context vector with LSTM outputs, predicting the next word via dense layer with softmax activation.
to generate two distinct latent representations, 2n and zy. The latent variables for each are generated from Gaussian distributions specified by the corresponding mean and variance values,
(Mz, log(o>)) Enncoder (x.,
(1)
2x NW 22), (My, (og(o;)) = Encodero ())
(2)
(3)
2y NN
(4)
Where & represents the encoder's variational parameters, this process generates pair of latent representa- tions (2z, zy): which are then used to perform a controlled interpolation to synthesize transitional weather images. The SC-VAE facilitates a stochastic mapping from the input images to their latent representations. The decoder, acting as generative model, utilizes an interpolated latent representation and a blended condition vector synthesize new images. Specifically, for a given pair of latent representations (2 (y)) and their corresponding condition vectors (1" 1)), the interpolated inputs calculated as follows,
fgg + -B-)a
(5)
1 31 + (1 3)1",
(6)
where 3 E [0, 1). The decoder then uses these interpolated inputs to generate a new image:
a Decoderą(z,
(7)
Figure 4 illustrates the fundamental process of latent data interpolation, which central to creating the transitional states of dataset. This process enables the generation of an interpolated image that smoothly blends the weather characteristics of the input pair (t). The parameter / provides fine-grained control
9

---

Published in Transactions on Machine Learning Research (01/2026)
Input image pair & One-hot encoded lable vectors
i
Interpolated image
- 2
+
C i
Zi
V
Z
L
B
+
Yi
j
i
Figure Latent space interpolation using SC-VAE. The encoder takes an input image pair (Ii, 36) and their corresponding one-hot encoded label vectors (1 ( to produce distinct latent representations (zi, 2z). By linearly interpolating the latent and label vectors, generate new blended latent vector and new blended label vector. The SC-VAE's decoder then uses these to synthesize novel image that smoothly blends the characteristics of the original two conditions.
over the relative intensities of the two weather conditions, which is a key advantage of our conditional approach. The joint distribution learned by the SC-VAE is now conditional on the input data and its associated label. The joint distribution is given by pa (á, z [), which is the product of the conditional prior distribution p( I) and the conditional distribution p( z, 2, The encoder learns the approximate posterior distribution (= x, ().
To enable backpropagation through the stochastic sampling process, our SC-VAE employs the reparameter- ization trick, expressed as
where is drawn from standard normal distribution with mean 0 and identity covariance matrix I.
The objective function is designed to minimize the reconstruction loss and the Kullback-Leibler (KL) diver- gence. The total loss is given by:
C(x, 2, 1) Crecon (2, (2, (KI(av(2 2,() |l p[= ()).
(9)
More specifically, the loss is defined as:
C(x, 1,,, 1, V, 1) =11 + 112 (KT.(u0 1. p(= ()),
(10)
where & = Decoder4 (2, is the reconstructed input. The KL divergence term encourages the learned latent distribution to align with the conditional prior distribution, ensuring the latent space is well-structured and disentangled based on the weather conditions.
In summary, we address the critical lack of transitional weather data by introducing a novel data generation methodology. We feed image pairs representing distinct weather conditions into our SC-VAE and perform controlled interpolation in the latent space. This process enables us to synthesize transitional weather images with varying intensities of fog, rain and snow. For example, when inputting sunny and foggy images, the SC-VAE's encoder generates two distinct, low-dimensional latent embeddings. The decoder then utilizes a blended latent embedding and a blended condition vector to generate images showing a smooth, data-driven weather transition between the original conditions. This novel process enables us create AIWD16-text, a unique dataset that facilitates the training of our model on diverse and transitional weather scenarios. By explicitly isolating and controlling weather effects, this approach leads to more robust image captioning and VQA performance across various weather conditions, which is crucial for autonomous driving applications.
4.2 VLAAD-TW
Figure illustrates the VLAAD-TW framework, which integrates a scries of components effectively gen- erate textual descriptions and answers to questions from images. consists of a Feature Encoder for Tran- sitional Weather (FETW), built. lightweight MobileNetV2 backbone, that extracts high-level visual
10

---

Published in Transactions on Machine Learning Research (01/2026)
features resilient to weather degradation. A Spatiotemporal Contextual Aggregator (SCA), comprising series of LSTM networks, processes the input text into coherent, context-aware representation. A Selective Attention-Guide Fusion Module (SAFM) dynamically weighs and fuses the visual features from the FETW and the linguistic context from the SCA. Finally, a Semantic Text Generator (STG), consisting of LSTM layers, synthesizes the fused information to produce a refined and context-aware textual output.
Feature Encoder for Transitional Weather (FETW). Let a weather transition sequence of length T, each image is processed by our FETW. This component, built on lightweight MobileNetV2 backbone (Sandler et al. 2018), is designed to extract high-level visual features that are robust against the degradation caused by dynamic weather. It outputs a sequence of high-dimensional visual embeddings (V1, V2...., Vr); capturing salient spatial and semantic information from each frame. These features are then fed into a series of dense layers that project them into a dine-simmnsional, task-specific representation, aligning their dimensions with the expected input of the Semantic Text Generator (STG). This transformation facilitates a seamless integration with the text generation modules by ensuring cach output token is conditionally grounded in a robust. visual context. To enhance the model's resilience and prevent overfitting to specific noise patterns, dropout is employed during training, encouraging the model to learn more generalized and diverse fcature representations.
Spatiotemporal Contextual Aggregator (SCA). This component processes the input text, such as a question or a prompt, to generate a coherent linguistic representation. It first tokenizes the input text into smaller units, with each token mapped to an integer from a predefined vocabulary. These indices are then transformed into dense vectors via an embedding layer, which captures semantic relationships between words, The embedded tokens are subsequently processed by an LSTM network, which is responsible for modeling long-range dependencies and maintaining robust contextual state across the sequence. For an input text of length T, the LSTM outputs hidden states (h1, h2...., h which serve as context-aware representations for the downstream modsl fusion. This process is crucial for providing the linguistic information necessary for the SAFM to interpret the visual scene accurately.
Selective Attention-Guides Fusion Module (SAFM). Our atlention mechanism, the SAFM, is specifi- cally designed to enhance the interaction between the visual and linguistic modalities. It calculates attention scores 01, 02.0..... . aN through dot product between the hidden states of the SCA and the image features extracted by the FETW. These scores dynamically weigh the relevance of specific visual features for each word in the generated sequence, with higher scores indicating a stronger focus on the most important image regions. Using these scores, the module computes a context vector Voontext as a weighted sum of the image features, Vcontext = E ViV,, where V/ represents the i-th image feature from the FETW. The context vec- tor encapsulates the most relevant and salient image information, effectively guiding the model to generate accurate and contextually appropriate descriptions even in the presence of weather-related visual degrada- tion. This cross- modal fusion is a crucial step that distinguishes our approach from models that process visual and textual information in isolation.
Semantic Text Generator (STG). The STG is responsible for synthesizing the final textual output. It processes the context vector Voontext, which is the fused output from the SAFM, alongside the output of the LSTM from the SCA. This combined input ensures that the text generation is grounded in both the most relevant visual fcatures and the overall linguistic context. An additional LSTM layer employed within the STG to process this combined context vector and the previous decoder output. This layer maintains the contextual flow of the generated text, ensuring it remains coherent and semantically relevant to the dynamic visual input. Finally, a dense layer with softmax activation function is applied to the LSTM's output. This produces a probability distribution over the entire vocabulary, enabling the prediction of the next word in the sequence based on a rich, multimodal understanding of the scene.
Rationale for VLAAD-TW Design. The design of our proposed VLAAD-TW framework is not a straightforward combination of existing components but a deliberate selection of architectures tailored to the unique challenges of real-time autonomous driving under transitional weather conditions. We employ lightweight MobileNetV2 backbone for visual fcature extraction within our FETW module. This choice is predicated on its highly efficient architecture, which utilizes depthwise separable convolutions and an inverted residual structure to deliver competitive performance with minimal computational requirements.
11

---

Published in Transactions on Machine Learning Research (01/2026)
Table 3: Overview of the dataset and implementation details for the proposed VLAAD-TW approach. + Batch size, - Input size, Op- + Optimizer, and Ir Ir--nnannnng rate.
Tisk
No of transition
states
AeWDi--txt Dataset
length (1)
Suquanee
No of sequences
No of annotations 45,450
Parkhone
Method
Data Generation
Eguche
Parameters
Training time (sec)
Backbone
Vision Language Assistance (VLAAD TW)
Eguehee
Parameter
time (sce)
Teatning
luage aptooiii
VIAA
16
1088
00
SCNAR
Lanei dats nnepplattoon
500
Rostetroction D:15
D10001 Up: Pali
KT doorggao
1000
Mokkieha +1STM
50
1:224
Lone Chose
00001, Oy Julm..
0.64
5839
2013
This efficiency is critical for meeting the real-time constraints of autonomous vehicles. While other feature extractors like VGG16 and ResNet are powerful, their computational intensity and larger model sizes render them unsuitable for this application. The MobileNet design is particularly effective at extracting salient, high-lovel features resilient to the visual degradation inherent in transitional weather. Our SCA module is built upon LSTM networks for text processing and sequential context modelling. LSTMs are uniquely suited for this task due to their ability to model long-range rangrrange dependencies and maintain robust contextual state over time, which is essential for both image captioning and VQA. They effectively overcome the vanishing gradient problem that limits the performance of standard RNNs on longer sequences, ensuring the model can maintain a coherent understanding of a full sentence or a series of frames. While GRUs offer a more lightweight alternative, their reduced flexibility makes them less suitable for the intricate context modeling roquired to generate procise and detailed textual descriptions of dynamic weather scenarios. The detailed computational and performance analyses of these architectural choices are provided in the next section.
5
Experiments
5.1 Dataset and Metrics
Table 3 summarizes the AIWD16-text dataset, outlining the transition states, annotation details, and im- plementation settings used for data generation and VLAAD-TW, including backbone, methodology, number of epochs, runtime, and hyperparameters. Transitional weather images for the AIWD16-text dataset were generated by a stochastic conditional VAF (SC-VAF), which was utilized to process pairs of input images with distinct weather conditions and perform latent data interpolation. Figure 5 shows a smooth latent representation of SC-VAE when presented with various extreme weather conditions, visually confirming its ability to learn a continuous and structured latent space. These input image pairs were collected from the Weather Driving (WD) dataset, including five weather conditions, sunny, cloudy, rainy, foggy, and snowy. Cloudy images were sourced from Cityscapes (Cordts et al., 2016), rainy images from RainCityscapes (Hu et al.) 2019) & MultiWeatherCity (Musat et | 2021), and foggy images from Foggy Cityscapes (Sakaridis et al. 2018). Snowy images are generated using ControLNet (Zhang et al.) 2023a), a stable diffusion-basse model. After addressing the class imbalance, each weather class in the WD dataset contains 293 images, re- sized to a resolution of 512 x 256 pixels. The final AIWD16-uext datasel comprises 4,688 weather transition sequences, each uniformly sampled at a length of = 10 frames, covering sixteen distinct weather transition states, namely, cloudy to rainy (CR), sunny to foggy (SF), sunny to rainy (SR), cloudy to snowy (CSn), cloudy to foggy (CF), snowy to rainy (SnR), snowy to foggy (SnF), foggy to rainy (FR) and vice versa. The dataset's total size is 7 GB. Figures G 7 g and present the qualitative results of the generated transi- tional weather sequences. The proposed VAE hesthss smooth progressions between distinct weather conditions (e.g., cloudy, rainy, foggy, snowy, and sunny), demonstrating its ability to capture continuous and realistic visual transitions across a specified sequence length T.
Dataset annotations. Our dataset annotation process encompassed two primary tasks: image captioning and visual question-answering (VQA). We manually created textual descriptions for the image captioning component that captured both environmental and traffic elements. The captions specifically documented current weather conditions and their intensity levels while also describing comprchensive road scene infor- mation, including moving vehicles and pedestrians in front of the ego vehicle and vehicles parked along the roadside. Also, we developed a VQA dataset by creating five carefully crafted open-ended question-answar pairs for cach image. These questions were designed to probe various aspocts of the road scene, focusing on key information such as the current weather and its intensity, the number and types of moving and parked vehicles, and detailed pedestrian-related in information.
12

---

Published in Transactions on Machine Learning Research (01/2026)
Rainy
Cloudy
Sunny
Foggy
-6
Figure Latent space representation of the input images generated by C-VAE. The representation is smooth and continuous.
T
Cloudy
Cloudy to Rainy
Rainy
Foggy
Foggy to Sunny
Semny
Sunny
Sunny to Rainy
Rainy
Cloudy
Cloudy to Foguy
Foggy
Figure Qualitative results of SC-VAE generated transitional weather sequences (c.g., cloudy rainy, foggy to sunny, sunny to rainy, and cloudy to foggy). For given length T, the model produces smooth and realistic transitions between weather conditions.
13

---

Published in Transactions on Machine Learning Research (01/2026)
3
Clondy
Cloudy Snowy
Snowy
Fopey to Rainy
Rainy
Snown
Showy Rainy
Rainy
Fougy
Foggy to Snowy
Snowy
Qualitative results of SC-VAE generated transitional weather sequences (e.g., cloudy to snowy, foggy to rainy, snowy to rainy, and foggy to snowy). For a given length T, the model produces smooth and realistic transitions between weather conditions.
Figure 7:
2
1-1
-
T
Rainy
Rainy to Snowy
Snowy
Sunny
Kainy
Rainy to Clondy
Clondy
Rainy
Rainy to Sunny
Suany
Qualitative results of SC-VAE generated transitional weather sequences (e.g., rainy to snowy, sunny to foggy, rainy to cloudy, and rainy to sunny). For a given length T, the model produces smooth and realistic transitions between weather conditions.
Figure 8:
14

---

Published in Transactions on Machine Learning Research (01/2026)
1=
Snowy
Snowy to Cloudy
Clondy
Rainy
Rainy to Foegy
Fogzy
Snowy
Snowy Foggy
Foggy
Voggy
Foggy to Cloudy
Cloudy
Qualitative results of SC- VAE generated transitional weather sequences (e.g., snowy to cloudy, rainy to foggy, snowy to foggy, and foggy to cloudy). For given length T, the model produces smooth and realistic transitions between weather conditions.
Figure 9:
Evaluation metrics. To evaluate the performance of image captioning and VQA tasks compared other methods, use the BLEU (Papineni et al. 2002) and ROUGE (Lin 2004) scores. The BLEU score measures the prccision of word sequences (n-grams) that match the generated text and ground truth. It is computed
as
BLEU = - exp
Σ
log P: 1=1
Where BP is brevity penalty and p; is the precision of é-grams (in this work BLEU-4 i.c., i=4). ROUGE-L evaluates the semantic similarity between the generated and ground truth texts using the longest common subsequence, and it is computed as.
ROUGE-L (2) Recall Precision
22. Rocall + Precision
Recall is the fraction of the ground truth tokens the predicted text. Precision the fraction of the predicted text's tokens in the ground truth, and 7 the weighting factor (default is 1) to balance the importance of recall and precision. Here, ROUGE-1 (L=1) measures the overlap of unigrams (single words) between the generated and ground truth texts, and ROUGE-2 (T=2)) measures the overlap of bigrams (two consecutive words).
In addition, to evaluate the quality the generated images, we use the Inception Score (IS). We also use Peak Signal-to-Noise Ratio (PSNR) and Signal-to-Noise Ratio (SNR) quantify the noise levels within the datasct.
Evaluating AIWD16-text dataset quality. To ensure that the AIWD16-text dataset reflects real-world scenarios, evaluate image quality using three key metrics: Inception Score (IS), Signal-to-Noise Ratio (SNR), and Pcak Signal-to-Noise Ratio (PSNR). Table 4 presents a comparative analysis of these metrics across benchmark datasets. The results show that AIWDI6-tex! consistently achieves higher IS, PSNR, and SNR values, highlighting its superior image quality and reliability as a resource for vision-language tasks, particularly under adverse weather conditions. While the generated transition images are generally effective, certain transitions such as snovy-rrain, rainy- sunny->raing and rainy rainy-rsunny, exhibit room for improvement. To further assess perceptual quality, conducted small-scale study (N 50), in which
15

---

Published in Transactions on Machine Learning Research (01/2026)
Table 4: Quality of the AlWett-text images measured using is, PSNR, and SNR metrics. Not appli- cable. Bold indicates the best values.
Dotaset/ Metrios
BDDTOOK Touct
Coedts
Musst
MW City
2011
CF
R5a
RF
18
5.00 4.51
6.13
51.70
58.66
19.39
$1.91
58.25
18.17
50.44
51.11
30.65
58.55
16.55
45.31
66.31
17.35
52.47
51.92
PSNR
27.73
51,59
12.30
50.13
S1,07
32,02
$1.22
$1.51
51.42
SL.15
90.52
92.55
30.11
30.32
20.96
50,11
SL.77
31.54
SNR
12.75
$4.20
83,37
34.25
$4.21
58.97
81.87
52,66
A1.22
81.30
31.10
31.88
50.15
50,95
81.22
81.01
Realisun
6)
4.63
4.6
4.23
4.55
3.80
0.52
4.30
4.02
3.66
8.87
8.33
3.24
3.76
3,67
3,00
3.0L
Table 5: Comparison of the AlWD16-text dataset with other AVT datasets. P+errett x-tNot present,
1 Not applicable.
No. of images
No. of captions
Image captioning
Adverse weather
Transitional
Datasct
VQA
weather
BDD-X (Kim et al. 2018
6,984
26,539
X
HDD
RRaaaniskka et al. 2018
7,744
x
X
CAP-DATA Fang et al. 2022
11,727
X
X
T2C (Deruyttere et
2019
850
11,959
V
X
X
DRAMA (Malla al.
2023
17,758
17,066
X
NuScenes- QA QQaa et
2024
1,000
34,000
X
X
DriveLM (Sima ct al. 2025
4,072
3,77,983
X
X
/
VLAAD Park et al.
2024
10,379
64,000
X
X
LIWD16-tex (Ours)
46,880
2,34,400
/
V
participants rated the realism of generated transitions on a 5-point Likert scale. The results (mean score 3.90) indicate that most transitions were perceived as visually realistic, though a few (including rain snow) received slightly lower ratings.
Software and machine setup. We used Python 3.8, PyTorch, and TensorFlow for our experiments NVIDIA Tesla M60 8GB GPU. Our VLAADD-W model, trained for 50 epochs with batch size of 64 and a learning rate of 0.0001, minimizes the cross-entropy loss function using the Adam optimizer. Also, our SC-VAE generation model trained for 100 epochs with a batch size and a lcarning rate of 0.0001 minimizes the KL divergence loss ionction using the Adam optimizer. We randomly split the dataset into 70% training. 10% validation, and 20% testing sets.
5.2 Comparison with Existing Vision Language Models.
Table 5 compares the proposed AIWD16-text dataset with other stato-of-the-art VLM benchmark datascts for AVT. As observed, only few datasets (Kim et al. 2018 Malla et al. 2023 Ramanishka al. 2018
Fang et al. 2022 include images featuring adverse weather conditions relevant to AVT. Notably, DRAMA
(Malla et al. (0223) and BDD-X (Kim et al. 20188) provide annotations related explicitly to adverse weather. However, the AIWD16 text dataset stands out as the first to feature transitional weather conditions, with
images annotated for both VQA and captioning tasks, addressing a significant gap existing benchmarks.
Tables and 7 demonstrate the performance comparison between VLAAD-TW and state-of-the-art VLMs the AIWD16-text dataset for image captioning and VQA tasks. We have selected these benchmark models for their diverse architectural approaches and proven capability in handling transitional weather conditions, Our analysis shows that VLAAD-TW achieves superior performance with a 2.01% improvement in BLEU score & 0.62% ROUGE-L metrics for image captioning and a 0.6% improvement in BLEU score & 0.83 in ROUGE-L for the VQA task, despite using significantly fewer parameters. The performance improvement stems from VLAAD-TW's lightweight architecture, which efficiently captures patterns in transitional weather conditions. In contrast, other foundation models exhibit lower performance due overfitting on our dataset, highlighting their limited generalization in this context.
16

---

Published in Transactions on Machine Learning Research (01/2026)
Table Performance comparison between the proposed and seate-of-the-art models for the image captioning task on the AIWD16-tex text Datasct. Bold indicates the best values.
Model
BLEU-4
ROUGE-1
ROUGE-2
ROUGE-L
BLIP (Li et al.
2022
39.52
38.63
27.34
37.55
BLIP-2 [i et al.
2023
43.69
44.95
31.93
43.10
EM VLM4AD base
(Gopalkrishnan et al.
2024
38.54
39.23
26.66
38.88
EM VLM4AD Q large
Gopalkrishnan (Sima et al. 2025
2024
43.59
45.61
28.79
43.15
DriveLM- M-Agent
45.12
47.36
29.55
42.31
VLAAD-TW (Ours)
47.13
49.29
31.99
43.77
Table 7: Performance comparison between the proposed and state-of-the-art models for the VQA task on the ATWD16-lext Dataset. Bold indicates the best values.
Model
Acc.
BLEU-4
ROUGE-1
ROUGE-2
ROUGE-L
BLIP Li et al.
2022
49.76
34.44
36.05
24.34
30.55
BLIP-2 2
Li et al.
2023
55.21
41.23
42.68
29.18
38.40
EM VLM4AD bawe Gopalkrishnan et al.
2024
49.21
39.27
45.61
28.79
39.15
EM - VLM4ADD -argg
iGopalkrishnan et al.
2024
45.77
37.95
44.23
26.66
38.88
DriveLM-A Agent
[Sima al. 2025
52.45
39.99
43.67
27.45
38.69
VLAAD-TW (Ours)
55.90
41.83
45.80
29.87
39.98
Table presents the performance of VLAAD-TW, pretrained on the ATWD16-text dataset and fine-tuned Drive-LM. The results demonstrate the effectiveness of VLAAD-TW, achieving competitive BLEU and ROUGE scores the Drive-LM benchmark.
Despite significantly fewer parameters. This fine-tuning performance of VLAAD-TW on the DriveLM dataset indicates that pretraining on the proposed AIWD16-text dataset effectively enhances the model's visual-language understanding and transferability. This demonstrates that the learned representations gen- eralize well beyond the pretraining domain, improving performance on downstream driving-related tasks.
Table presents the performance of VLAAD-TW, which is pre-trained on the AIWD16-text dataset and subsequently fine-tuned the nuScenes dataset to evaluate its transferability and robustness VQA task. The results show that VLAAD-TW basclines, including OmniDrive, across multiple metrics such as accuracy, BLEU, ROUGE-L and CIDEr scores. These findings demonstrate that VLAAD-TW pretraining on AIWD16-texl elfectively enhances the model's ability to adapt to diverse and evolving driving scenarios, thereby validating both the effectiveness of VLAAD-TW and the practical utility of the AIWD16-text dataset. Computational analysis. We evaluated the memory and computational efficiency of VLAAD-TW, crucial factors for real-time systems with resource constraints. Table 10 compares computational complexity metrics between VLAAD-TW and
achieves notable improvements over existing
Table 8: Performance comparison of proposed and state-of-the-art models on the DriveLM VQA task. Bold indicates the best values.
Model
BLEU-4
ROUGE-1
ROUGE-2
ROUGE-L
BLIP
(Li et al.
2022
34.44
36.05
24.34
30.55
BLIP-2
[ et al.
2023
41.23
42.68
29.87
38.98
EM - VLM4AD buse
iGopalkrssha
et al.
2024
45.36
57.77
29.11
71.98
EM VLM4ADo -large
Gopalkrishnan
2024
40.11
55.91
27.85
70.72
DriveLM-Auee Agentt
Sima et al. 2025
53.09
56.11
28.99
66.79
VLAAD-TW (Ours)
53.81
58.04
29.15
72.45
17

---

Published in Transactions on Machine Learning Research (01/2026)
Table 9: Performance comparison of proposed and state-of-the-art models pretrained on the AIWD16-text dataset and finetuned on the NuScenes Datasct. Bold indicates the best values. >o=Noot applicable.
Model
Acc.
BLEU-4
ROUGE-L
CIDEr
Omni-L
Wang et al.
2025
53.27
42.45
61.55
73.89
Omni-Q
Wang et al.
2025
52.15
68.86
OPT Zhang et al.
2022
38.80
47.70
EM VLM4AD bose
Gooaalrishhan et al.
2024
42.36
68.99
LLAMA- Addtter
Zhhag et. al.
2024b
43.17
67.56
VLAAD-TW (Ours)
57.96
45.73
69.09
75.62
Table 10: Computational comparison between the proposed model and statc-of-thc-art VLMs.
Not
applicable, Bold indicates the best values.
Vision Backbone
Text Backbone
No. of Param
Memory (GB)
Time
Model
FLOPs
FPS
(ms)
EM VLM4AD
Copalkrishnau et al.
2024
VIT
T5-Base
235M
9.47B
0.94
2.9
635
EM VLM4ADo Gopa Iriis hnan et
2024
VIT
T5-Large
769M
31.5B
0.77
1.2
833
DriveLM-Agent Sima
LoRA, BLIP-2
3.96B
439B
14.43
0.16
6250
LLM-Drriee
Chen et al 20246
LLaMA-7b
7B
268B
28
BLIP et
2022
VIT
BERT
300M
4.2B
0.95
BLIP-2 et
2023
CLIP VII
Frozen LLM
4.1B
23B
15
2.1
VLAAD-TW (Ours)
CNN
LSTM
12M
810M
0.25
11.2
89.3
state-of-the-art VLMs for AVT across five key indicators: model parameters (param), FLOPs, memory utilization, FPS and inference spoed (time). VLAAD-TW demonstrates remarkable efficiency, requiring only 12M parameters, 810M FLOPs, and 0.25GB of memory. Compared to the current. foundation VLMs, this reflects 85% reduction in parameters, 80% reduction in FLOPs, and 73% reduction in memory usage. These significant improvements streamline the architecture, enable faster inference and higher frames per second (FPS), and greatly reduce resource consumption while maintaining competitive performance. Our model achieves substantially higher FPS and inference speed compared to other models. Such optimizations make VLAAD-TW particularly well-suited for resource-constrained environments and real-time applications where rapid inference and minimal resource consumption are essential.
Qualitative analysis. Figures TO and 11 show qualitative results for image captioning and VQA across sunny to foggy, cloudy to rainy, sunny to rainy weather transitions, highlighting VLAAD-TW's ability to interpret scenes and generate accurate captions and answers under challenging conditions. Despite strong overall performance, certain limitations were observed. The model encounters difficulties perceiving small and distant road elements during moderate to heavy rain and fog, primarily due to varying and low illumination conditions. Additionally, the model occasionally misinterprets subtle weather variations. These challenges will be addressed in future work.
6 Ablation Study
Model performance across various transition states. Table I1 summarizes the image captioning performance of the proposed VLAAD-TW model across various transitions in the AIWD16-text dataset. The Rainy to Cloudy, and Cloudy to Rainy transitions yield the best results. Similarly, Table 12 presents the VQA performance, where these (ransitions outperform others. We performed an additional ablation study to assess the impact transitional weather modeling by comparing VLAAD-TW trained on (a) WD (Weather Driving) alone and (b) ATWD16-text. The results in Table show that training with ATWD16- text leads substantial improvements across all evaluation metrics, indicating that transitional weather data provides more informative supervision.
18

---

Published in Transactions on Machine Learning Research (01/2026)
weatherr sunay van moving and ears parked at left eyelists moving G: Weather sunny, A moving and cars parked left side 2 eyeelists moving right side
P: Weather light fog I van moving and 2 eyelists right G: Weather light fog. A van is moving and 5 cars parked at left side eyelists moving right side
P: Weather is moderate fog cur mosing curs parked G: Weather is moderate fog. A van is moving and cars parked at left side eyclists moving right
Weather heavy fag car moving and cars parked at left I cyclist at right G: Weather heavy and cars parked side 2 cyclists moving right side
curs parked
cyclists right
moving
P: Weather cloudy cars msvving surrounded by cars parked pedestrians walking left G: Weather is Cloudy.A cur is moving apposite mosing surrounded by parked curs same direction pedestrians standing left
Weather moderate rain.
Weather moderate
P: Weather heavy ruin.
cars moving surrounded by curs parked pedestrians walkknng left
6 cars moving surrounded by
moving surrounded
cars parked 4 pedestrians at left
cars parked pelestrians
Weather light
moving
Weather moderate rain.
G: Weather heavy rain, A car
opposite 4 cars moving surrounded
moving opposite 4 cars moving surrounded
moving opposite moving surmundeed by parked cars same direction pedestriaus standing at left
parked in direction
same direction by podestrians standing lc0
pedestrians standing lefe
P: Weather is sunny, 3 cars opposite
Weather moderate rain. moving same direction cars oppostle G: Weather light rain. 5 cars moving in direction cars apposite
P: Weather is heavy rain. 3 cars opposite cars moving same direction G: Weather moderate rain. curs moving in the sume direction and opposite
P: Weather heavy rain. moving sume direction cars opposite Weather heavy raia. cars moving same direction and cars opposite
direction G: Weather is sunny. cars muving in the same dirvetion and 3 upposite
Figure 10: Qualitative analysis of the proposed VLAAD-TW approach for image captioning on the AIWD16- text dataset. Green highlights the ground truth, Blue highlights the predicted text, and Red highlights the wrong prediction. Q-Query, P P-+Precictio,, G-Grround truth.
19

---

Published in Transactions on Machine Learning Research (01/2026)
Q: What is the current weather scenario?
Sunny Sunny
Q:
woother vcenario?
What scenarfo? Light log C: Moderate fog Q: Are there any pedestrians fror of ego chick Pt No Gr No
Q: What the curreal weather scenario? Hervy fog there any parking vehicles Pt cars parked
A van is moving cars parked left road side and syclist right
Q: Deseribe the current scene?
Q: What are the moving objeccs infrost of vehicle
Light fog G: Light fog
G: A van is moving and 5 cars parked right side
P: moving
G: cars parked lefaide
left side eyelist
Q: What is the corrent weather scenario?
Q: What the current weather sccuario?
Q: the current waater scenarto?
What is the currest weather scenaris? Mederate
Clondy
Heavy G: Henvy rain
Q: Deseribe the current scene!
P: Moderate rain Light ruin
Mederate
Qr Are there any parking vehicles P: 5 cars parked at leftaide cars at right
mon ling sarrounded by parked
Q: What are the moving objects infruat ego vehicle
Q: Are there any pedoutrians infront of ego vehicle
pedestrians walking at left
GIA car moving apposite cars moving
and pedestrians
P: podestriam C: 4 podestrians
G: 6 cars parked leftside and
surrounded by parked cans same
G:
few parked right
rectien 4 pedestrians danding at lsk
Q: What the Sunny G: Sunny
Q: What weather sconarto?
What current weather scenarie?
Q: What the current wenther scenarto?
Q: Describe the current scene?
P: Light rain G: Light rain Q: What are the moving objects Infrent vehicle
P: Heavy rain G: Madurate rain
P: Heavy minn G: Henvy rain Q: Are there parking vebicles Pt parked
P: cars upposite cars moving dirretion
Q: Are there polostraas infront of vehicle No
direction oppreite
cars moving
moving
Figure 11: Qualitative analysis of the proposed VLAAD-TW approach for VQA on the AIWD16-text dataset. Green highlights the ground truth, Blue highlights the predicted text, and Red highlights the wrong predic- tion. Q QuQuery, P P-+Prediction, G-+rround truth.
20

---

Published in Transactions on Machine Learning Research (01/2026)
Table 11: Performance of the proposed model for image captioning on each transition state of the AIWD16 dataset. Bold indicates the best values.
Transition/
FS
C55
SnC
1t5m
IFEL
BLEU-4
40.24
50.07
46.19
47.91
38.27
58.3 38.10
46.12
47.41
43.29
42.01
35.33
37. 17.89
40.65
40.02
45.22
47.40
ROUGR-1
51:30
54.45
50:10
18. 05
11:33
60 am
50:15
18.81
66-11
45. 86
40-70
10. 10-23
41:95
43.85
50-16
19-54
ROUCE-2
88:16
89.28
34.44
82. 81
23.67
27
36.28
$6.96
33.05
38.10
27.44
27. 01
29.78
81. 31.46
57.38
87:16
17:16
48.07
39.78
37:
30.12
28
47:12
42.02
42
29.79
37:15
Table 12: Performance of the proposed model for VQA on cach transition state of the AIWD16-text datasct. Bold indicates the best values.
Transition/
Ancuraar
58.67
59.65
19-90
18: 76
12:17
81. 05
56.76
55.25
15.77
63 38
42:11
13. 13.23
45.60
45-77
49.33
47:37
BLEU-4
45.80
47.92
44.45
42:24
85.70
96
44.15
43.26
39.19
38. 42
36.43
37.00
39.77
89: 99-900
43.15
42.94
ROUGGE-1
50.35
38
40:21
39.89
40.
45:08
Computational and performance analysis across various feature extractor backbones. Table details the computational complexity of the proposed VLAAD-TW using VGG16 (Smoonyan Zisserman 2014) and ResNet50 (He al. 2016) feature extractors. Compared to MobileNetV2, VGG16 requires 7.2x more memory, 19x more FLOPs, and 11.5x more parameters, while ResNet50 demands 4.2x more memory, 5 x more FLOPs, and 3.6x more parameters. The table also highlights the performance of these feature extractors, showing that VLAAD-TW with MobileNetV2 outperforms others in terms of BLEU and ROUGE scores for the image captioning task. Figure 12 provides a qualitative comparison of the proposed VLAAD-TW model across different feature extractors for the VQA. For each image-question pair, answers produced by VLAAD-TW using various visual backbones choice of feature extractor influences semantic grounding, object recognition, and reasoning accuracy, Performance analysis across various sequence model backbones. Table 15 reports the performance of the proposed TLAAD-WW using three recurrent sequence models, GRU (Chung et al. 2014), RNN (Elman 1990), and LSTM (Hochreiter & Schmidhuber 1997). As shown in the table, the LSTM variant achieves the best overall performance among the evaluated models. This observation is further supported by the qualitative examples in Figure [3 where the LSTM- model provides more accurate and contextually appropriate answers. These results indicate that increasingly exprossive recurrent units enhance question understanding and multimodal reasoning in VQA. Overall, the analysis supports the design choice of pairing the VLAAD-TW architecture with efficient components such as the MobileNet V2 feature extractor and the LSTM sequence model.
shown to illustrate how the
Ablation of SAFM and SCA Modules. Furthermore, we conduct two modulc-level ablations to isolate the contribution of the higher-level components of our architecture. First, in the w/o SAFM variant, we remove the attention-guided fusion block and instead form global visual and textual descriptors using only the final hidden states of the image and question encoders. These vectors concatenated and passed through fully connected layer, thereby eliminating all cross-modal attention and fine-grained relevance weighting. Second, in the w/o SCA configuration, we disable the entire Spatiotemporal Context Aggregator and rely solely on single-pass text encoder to obtain the question representation, removing the additional temporal reasoning and contextual enhancement normally provided by SCA. These ablations allow us to
Table 13: Performance comparison demonstrating the benefit of training VLAAD-TW with AIWD16-ter rather than WD alone. Bold indicates the best values.
Datasct.
BLEU
ROUGE-1 ROUGE-2 ROUGE-L
WD
37.50
37.40
20.28
33.33
AIWD16-text
47.13
49.29
31.99
43.77
21

---

Published in Transactions on Machine Learning Research (01/2026)
Table 14: Computational and performance comparison of the proposed VLAAD-TW with different feature
extractors. Bold indicates the best values.
No. of parameters
Feature extractor
FLOPs
Memory
BLEU-4
ROUGE-1
ROUGE-2
ROUGE-L
VGG16 [Simonyan & Zisserman
2014
138M
15.5B
1.8
41.24
40.00
23.48
37.21
Resnet50 He el al. 2016
50M
4.2B
0.9
44.47
39.29
24.55
36.42
V2 Sandler et. al. 2018
12M
810M
0.25
41.83
45.80
29.87
39.98
What current
weather secnario?
current weather sernario?
Q: What the current weather scenario?
Q: What the current weather scenario?
MobileNetV2
on ruad and eyele parked,
G: A car moving at the front
Deseribe the currest semee?
P: Faggy G: Feggy
pedestrian walking
I car moving and pedestrians on the road G: A car is moving at the front and podestrians crossing the rad
O: What
P: snow G: Mixed fog and snow movlag objects
infront ege vehicle
P: 3 polestriam G: podistrians
Q: Are there any pedestriuns infront of ego whicle
Light G: Moderate
G: cycle parked at right side
P: Heavy G: Heavy snow there any parking vehicles Pt cycle parlied on rond
pedestriuns crossing the road and
cycle parked right side.
What curreat
weather scemario?
What the current scomario? P: fog G: Mixed fog and snow
Q: What is the curment weather semario?
Q: What is the current weather scenario?
P: Foggy G: Foggy
P: moving pedestrians and cycles parked front, and Gt cur moving the frone, d
Deseribe the current scene?
Q: W What are the moving objects
Pt
Mederate Q: there any pelrstrians Infrent of ego vehiile
Pt Heavy snow G: Heavy snow Are there any parking vehicles P: I evele front side parked right side
infrunt igo vhiile
VGG16
P: and pedestrians
road
P: 3 pedestrians G: 4 pedestrians
pedestrians crosing the road and
G: moving at frout and
cycle parked right
pedestrians crosing the road
Q: What current weather scenario?
Q: What is the current weather seenario?
What curreat
What the
ResNet50
P:A car froat eyclists moving on the road and cyele parked read. and
polestraas crosiing the rad and
GiA car moving the froat, 4
Describe curreat scene?
P: Fages G: Foggy
P: 1 car and eyclisss moving leftside
Mederate
O: What are the moving objects
infront of ego vehicle
moving front
Mixed
P: Moderate snow G: Maderate snow there any pedestrians
P: pedestrians G: pedestrians
infrent ego
weather scenario?
Are there any parking vehicles
P: Heavy snow G: Heavy snow
P: I eycle parked the road eycle parked right
weather sccmario?
cycle parked right
pedestrians crossing road
Qualitative evaluation showing how different feature extractors affect the performance of the proposed VLAAD-TW model. Green highlights the ground truth, Blue highlights the predicted text, and Red highlights the wrong prediction. Q-Query, P-Prediction, G-Grouund truth.
Figure 12:
22

---

Published in Transactions on Machine Learning Research (01/2026)
Table 15: Performance comparison of the proposed VLAAD-TW with different sequence models. Bold
indicates the best values.
Sequence model
BLEU-4
ROUGE-1
ROUGE-2
ROUGE-L
GRU Chunng 2014
38.39
22.23
30.61
RNN Elman 1990
28.65
36.21 45.80
18.99
26.78 78
LSTM (Hochreiter Schmidhuber 1997
41.83
29.87
39.98
Table 16: Ablation analysis showing the effect of removing the SAFM and SCA modules on AIWD16-text for the VQA lask.
Variant
BLEU-4
ROUGE-1
ROUGE-2
ROUGE-L
W/o SAFM
34.92
36.29
24.65
33.72
W W/o SCA
38.25
40.96
27.59
36.66
VLAAD-TW (Full model)
41.83
45.80
29.87
39.98
directly quantify the contribution of SAFM and SCA to robust seene understanding and descriptive quality, as summarized in Table 16 and illustrated qualitatively in Figure 14
Generalization of VLAAD-TW on unseen weather domains. Table 17 reports the performance of VLAAD-TW on unseen weather domains. The model was trained on the Cloudy to Rainy transition and evaluated other transition scenarios. A noticeable drop in performance highlights the domain gap between training and testing conditions. Figure 13 further illustrates this effect by showing qualitative results when the model is trained on the AIWD16-text dataset and tested on the unseen BDD100K dataset. As ground-truth VQA annotations are unavailable for BDD100K, the incorrect predictions were identified through visual inspection. We plan to address this domain discrepancy in future work using unsupervised domain adaptation techniques.
6.1 Discussion on Edge Cases
Unusual weather patterns. The training data. incorporates transitional weather scenarios of varying intensity to strengthen model robustness across conditions. Using the SC-VAE, we perform controlled data interpolation with the parameter & E [o, 1] to generate a continuous spectrum of weather transitions, from to This methodology allows us to synthesize an extensive datasel that captures various weather variations. This diverse training data enables our model adapt effectively while sustaining high performance across wide range of scenarios. However, ensuring optimal performance in highly unusual unforeseen weather patterns remains challenge, presenting a direction for future work to further strengthen VLAAD-TW's adaptability and ensure reliable performance across wider range of environmental conditions.
Performance on real-world noise and adversarial robustness. Our model performs well across various challenging weather conditions, and real-world noise and adversarial perturbations may introduce additional challenges. Weather transitions often exhibit irregular patterns, where certain dependencics are more critical than others. To further strengthen robustness, we plan to explore adversarial defense strategies, including
Table 17: Quantitative results of the proposed VQA model trained on the Cloudy to Rainy transition and tested on other weather transitions evaluate generalization to unseen weather domains. Bold indicates the best values.
Transition Matric
RCC
SF
FS
SR
R1
CF
FCI
c
SaC
SnR
RS5
SoFF
FSn
FR
RF
Accuracy
49.12
42
88
22.22
37.42
36.
40.37
BLEU-4
ROUGE-1
34.60
56.56
ROUGE-2
ROUGE-L
26.66
32.97
23

---

Published in Transactions on Machine Learning Research (01/2026)
Q: What current wenther scenario?
What the scenario?
Q: What is the current weather secnario?
Pr. Clondy G: Cloudy
P: Light rain G: Light rain
Q: the weather scenaris?
Q: Describe the current scene? image shows parked and person on foolpailk,
Q: What are the moving objects Infront ego vehicle
P: Light rain G: Moderate rain
Q: there any parking vehilles
P: Hervy rain G: Heavy rain
LSTM
Pr No moving objects infrent of ego
Q: Are there any pedestrians Infrent vehitle pedestrian G: pedestrian
3 parked cars lefeside
G: The image shows 3 parked vebicles and I person on footpath
G: parked vehicles infrent of ego vehicle
G: No moving objects Infront of vehicle
Q: What is the current weather scenario?
weather scenario?
Cloud: G: Cloudy
Light Light moving objects
Q: What is the current weother scenario?
weather secuario.
What current
The shows vehicles road image parked vehicles person footpath
Q: Describe the carrent scene?
Infront of egu vehicle P: 2 vhhicles infrent of vehicle
Mederate Q: Are there any pedestrians infront of ego vehicle
Q: Are there parking vehicles velicles
Pt Moderate rain G: Heavy rain
GRU
parked vehicles infreat rgo vehicle
F: pedestrian G: pedestrian
moving objects infront
Q: What the
Q: What the current weather secnario!
P: Cloudy G: Cloudy
weather seenario?
P: Light rain G: Light ruin
Q: What is the current weather scenario?
What
P: Image shows there are 3 vehicles meving
Q: Describe the carrent scene?
Q: What are the moving abjects infeant
vehiele
P: Light rain Moderate rain podestrians ego
Q: Are there any parking vehicles P: 2 velieles parking on rend and parked vehieles infreat of ego vehikle
Moderate G: Heavy
pedestrains
Pt vehicles infreat ego vehicle
RNN
The image parked and I person on footpath
pedestrian G: 1 pedestrian
G: No moving objects infront vehicle
Figure 13: Qualitative evaluation of the proposed VLAAD-TW method using different sequence models on VQA examples. Green highlights the ground truth, Blue highlights the predicted text, and Red highlights the wrong prediction. Q-Query, P-Prredcction, G-Ground truth.
24

---

Published in Transactions on Machine Learning Research (01/2026)
weather scmario"
What
What current weather scemurio?
Q: What current weather secuario?
VLAAD TW (ours)
P:A Deseribe the current scene?
pedestrians walking road and cycle parked,
P: Feggy G: Fegny
P: snow G: Mixed fog and snow
What are the moving objects ego moving and pellestrians on road
weather scemarie!
Light G: Moderate any podestrians
P: Heavy snow G: Heavy snow there parking vehicles P: cycle parked
infront ego vehidle
podestrians crossing the read and eycle parked right side
G: A car moving the front
G: car is moving at the front and pedestrians crosinng the road
P: pedestriams G: pedestrians
G: cycle parked at right side
O: What the currest
Q: What is the current wetther scenaris?
wenther scenario?
Q: What current weather scenario?
Q: What current weather scenario?
P: Faggy G: Fogny
Mixed
Wio SAFM
on Inotpath and eycle parked road, and
P: car moving on read peolstrians
scone?
P: 1 car pedestrians on the footpathh
Infrent ego vehicle
the moving objects
P: I iight snow G: Moderate vnow Are there pedestrians infront vehicle pedestrians pedestrians
P: Heasy snow G: Heavy snow Q: there parking vehicles P: parked cycle parked rightside
G: A moving the front, d pedestrians crossing the and
G: car moving the front and 4
pedestrians cressing
cycle parked right
wenther secuario?
Q: What current weather scenario?
What the weather scemario?
Q: What is the current weather scenarie?
P: Faggy G: Feggy
P:A moving.4 pedestrians walking
Q: Describe the current scene?
snow Mixed fig and snow
moving ebjects
Light snow G: Moderate snow Are there any pedestrians infront of vehide pedestrians pedestrians
Hlavy snow Q: Are there any parking vehicles Pt eycle parked at rod G: cycle parked right side
Heny
Wio SCA
A car moving the frunt, pedestriaus crossing the road and cycle parked right side
eyele parked read. and
P: car moving 3 pedestriaas on the road G:A car moving the front and 4 pedestrians crossing the
infrunt of ego vehicle
Figure 14: Qualitative analysis showing the effect of removing the SAFM and SCA modules on AIWD16- text. Green highlights the ground truth, Blue highlights the predicted text, and Red highlights the wrong prediction. Q Q-tQuery, P~>Prediction, G-Grroud truth.
Q: What is the current weather scenario?
Q: What the current weather scenario? Light rain Q: Are there any pedestrians infrent of ego vehicle No
Q: What is the current weather scenario? P: Heavy rain
Q: What is the current weather scenario? Pt Light rain
P: Light rain seene?
Q: Deserthe the
Q: What are moving objects infroat of ego vehicle P: cars moving
Q: Are there any parking vehicles Pt 6 cars parked at road side
P: 6 cars are moving
is parked right
Figure 15: Qualitative analysis of the proposed VLAAD-TW model trained on the AIWD16-text dataset and tested on the unseen BDD100K dataset. The results illustrate prediction errors caused by domain discrepancies with visual inspection, as ground-truth VQA annotations are unavailable.
25

---

Published in Transactions on Machine Learning Research (01/2026)
adversarial training and uncertainty quantification techniques, to mitigate potential vulnerabilities. These enhancements will ensure our model maintains strong generalization across real-world weather conditions.
6.2 Potential Biases of AIWD16-text Dataset
While the AlWD16-text dataset provides diverse intensities of weather conditions, it has certain limita- tions and potential biases, which we intend to investigate further in future work. (i) Limited geographic representation: The dalaset contains static background scenes throughout weather transition sequence, representing only a limited geographic area. This design enables a controlled and interpretable setting for evaluating vision-language models and perception systems across weather transitions. By isolating weather variations from background motion, our method provides a reliable mechanism for analyzing model robust- ness to transitional weather. In future work, we plan explore video-based weather transition models to capture dynamic scene background along with weather transitions. (ii Temporal sampling bias: The dataset caplures transitions at 10 frames per second, which may not adequately represent all weather intensity vari- ations. This fixed sampling rate could introduce bias by missing intermediate weather states. In future work, we will use different sampling rates and create annotations for them. (wi) Potential biases in data division: The dataset is divided randomly into training, validation, and testing sets. However, this approach may not account for biases in the data distribution, such as variations in the number of images of various weather scenarios, traffic scenarios or road elements. (iv) Annotation biases: The accuracy and consistency of the annotations for visual interpretation could be subject to human biascs. This could lead to systematic under-representation misclassification of certain road elements, which may impact the model's ability to generalize to real-world scenarios. Evaluating the quality and potential biases in the road element textual annotations should be focus of future investigations.
6.3 Unique Advantages of SC-VAE Weather Transition Generation
While latent space interpolation known generative technique, our primary contribution lies in its novel and lask-specifie application to synthesize continuous and diverse range of transitional weather states. Existing generative methods often focus on discrete attribute manipulation or simple style transfer. In contrast, our approach models the scamless and progressive evolution between complex weather conditions, directly addressing the critical lack of continuous weather data for robust VLM training in AVT. This adaptation is not a trivial reuse of existing methods but a purpose-built, domain-aware methodology designed to solve a significant, real-world challenge in autonomous driving perception. To the best of our knowledge, this is the first work to successfully apply data interpolation to generate transitional weather data, making it distinct and timely contribution to the field.
7 Limitations and Future Work
Despite the effectiveness of the VLAAD-TW, the current work has certain limitations that we plan to address in future research.
1. Background consistency and other possible weather transitions. While the generated images depict realistic scenarios, the backgrounds remain static across weather changes due to the use of input image pairs with consistent backgrounds. This design enables accurate evaluation of vision-language tasks under varying weather conditions. Future work will explore transitional video generation with dynamic backgrounds while preserving fine-grained weather attributes.
2. Long-range dependency. While VLAAD-TW excels in real-time AVT tasks and outperforms other models, there remains room for improvement. SCA is built on top of an LSTM backbonc, which strug- gles with capturing long-range dependencies, limiting performance on complex weather sequences (sunny to rainy). Replacing LSTMs with Transformers could enhance the ability to model intricate temporal relationships while maintaining efficiency. Furthermore, the model may face challenges in generalizing transitional weather conditions and handling long and intricate sequences. Future work could focus on incorporating domain adaptation and optimixing the architecture for longer input sequences.
26

---

Published in Transactions on Machine Learning Research (01/2026)
3. Unusual weather scenarios. The dataset used to train VLAAD-TW includes a variety of transitional weather patterns to improve its performance in different conditions. These transitions, which have varying intensity levels over time, enable the model to better adapt to multiple scenarios. To further enhance the modcl's performance on previously unseen weather patterns, future work will explore using unsupervised domain adaplation.
4. Transition image quality. When transitioning between snowy and rainy conditions, image quality can suffer from distortion and noise due to the rain. To counteract this, future work will incorporate denoising algorithms as a preprocessing step. This will provide cleaner input images for the transition generation process, which should strengthen VLAAD-TW's ability to handle challenging weather shifts.
5. Enhancing robustness to adversarial attacks. High noise levels in the dalaset can compromise model robustness, making it less reliable in real-world situations, especially with natural noise or adversarial perturbations. Future research will focus on developing strategies for both adversarial attacks and defenses to make VLAAD-TW more resilient once it is deployed.
6. Physical plausibility. While our framework effectively generates controllable visual transitions across diverse weather conditions, certain transformations (e.g., rain + snow) may show limited physical plau- sibility. Future work will aim to enhance the physical consistency of these transitions by incorporating physics-informed priors and temporal regularization to achieve smoother and more realistic weather dy- namics.
7. Risk-aware vision-language reasoning. A promising and critical future direction for VLAAD-TW is the integration of real-time risk analysis into the VLM pipeline. While our current work provides seman- tic undorstanding of transitional weather scenarios, the next logical step is to move from description to proactive risk assessment. We propose extending our framework to localize potential hazards within the visual scene, such as slippery road conditions, obscured pedestrians, or hydroplaning risks. This would enable the model to generate not only descriptive captions but also actionable, risk-aware reasoning, such "Caution: The road ahead is wet and car is braking suddenly.' Coupling risk localization with con- textual explanation ions enables a more informed and interpretable basis for decision-making. Furthermore, incorporating causal reasoning into the risk assessment, to distinguish truc danger signals from weather- related visual noise, is vital for enhancing the safety, generalization, and trustworthiness of autonomous systems operating in these challenging dynamic environments (Malla et al. 2023 Zhang et al., 2023b).
8 Conclusion
In this paper, we propose VLAAD-TW (Vision-Language Assistance for Autonomous Driving under Tran- sitional Weather), a novel framework for image captioning and VQA in challenging weather conditions. VLAAD-TW introduces a novel cross-modal, spatiotemporal reasoning architecture specifically engineered for interpretable perception in autonomous driving under transitional weather conditions. It integrates Feature Encoder for Transitional Weather (FETW), built on an efficient. MobileNet backbone, for robust visual feature extraction. A Spatiotemporal Contextual Aggregator (SCA), based on cascaded LSTM net- works, processes input text to provide rich linguistic context. A Selective Attention-Guidde Fusion Module (SAFM) then dynamically weighs and combines information from both modalities. Finally, a Semantic Text Generator (STG) produces precise, context-aware text from this fused representation. Our key contribu- tions include a lightweight, computationally efficient architecture that significantly reduces model complexity compared to the baseline, achieving an 85% reduction in parameters, 73% reduction in memory requirements while maintaining competitive performance. Also, introduce AIWD16-text, a comprehensive vision-language dataset that features rich collection of images spanning sixteen distinct transitional weather states, generated using a Stochastic Conditional VAE (SC-VAE). The images are further enriched with manually annotated image captions and open-ended question-anse answer pairs, making ATWD16-text a unique and valuable resource for training and evaluating vision-language models in this underexplored domain. Extensive experiments demonstrate VLAAD-TW's superior performance, achieving 2.01% % BLEU and 0.62% ROUGE-L improvement in image captioning and 0.6% and 0.83% gains
80% reduction in FLOPs and a
27

---

Published in Transactions on Machine Learning Research (01/2026)
in VQA tasks while using significantly fewer parameters than existing methods. Notably, our model main- tains robust performance across transitional weather. Future work will focus on data generation in dynamic backgrounds by integrating moving scene information alongside variations in weather intensity.
9 Acknowledgement
This work was supported in part by the NMICPS Technology Innovation Hub on Autonomous Navigation (Ti- HAN) at the Indian Institute of Technology Hyderabad under Project No. TiHAN-ITTHO0S /000//2////00222255005
References
Jaeseok Byun, Dohoon Kim, and Taesup Moon. Mafa: Managing false negatives for vision-language pre- training. In Proceedings of the LEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 27324,27324, 2024.
Holger Caesar, Vikash Bankiti, Simon Hodge, Shenghong Liao, Xin Lyu, Vibhav Singh, Shaoshuai Xu, Zhiqiang Zhang, and Lijun Zhang. muscenes: A multimodal datasct for autonomous driving. In Proceedings of the IEEE/CVE CCVF Conference on Computer Vision and Pallern Recognétton (CVPR), pp. 1160 1070, 2019.
Chengtai Cao, Xinhong Chen, Jianping Wang, Qun Song, Rui Tan, and Yung-Hu Li. Sgdel: Semantic- guided dynamic correlation learning for explainable autonomous driving. In Proceedings of the Thirly- Third International Joint Conference on Artificial Intelligence, IJCAI-24, pp. 596 604, 2024.
Kai Chen, Yanze Li, Wenhua Zhang, Yanxin Liu, Pengxiang Li, Ruiyuan Gao, Lanqing Hong, Meng Tian, Xinhai Zhao, Zhenguo Li, et al. Automated evaluation of large vision-language models on self-driving corner cases. In 2025 IEEE/CVE Winter Conference on Applications of Computer Vision (WACV), pp. 7817-7826. TEFF, 2025.
Long Chen, Oleg Sinavski, Jan Hünermann, Alice Karnsund, Andrew James Willmott, Danny Birch, Daniel Maund, and Jamie Shotton. Driving with Nms:: Fusing object-level vector modality for explainable au- tonomous driving. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pp. 14093 14100. IEEE, 2024a.
Yangyi Chen, Xingyao Wang, Hao Peng, and Heng Ji. A single transformer for scalable vision-language modeling. Transactions on Machine Learning Research, 2024b. ISSN 2835-8856.
Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint arXiv:1412.355: 2014.
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understand- ing. In Proceedings of the IEEE Conference on Computer Vision and pattern Recognition (CVPR), pp. 3213 3223, 2016.
Thierry Deruyttere, Simon Vandenhende, Dusan Grujicic, Lue Van Gool, and Maric-Francin Moens. Talk2car: Taking control of your self-driving car. ar preprind 1rXic:1909.1082. 2019.
Jorge Luis Guevara Diaz, Bianca Zadrozny, Campbell Watson, Danicla Szwareman, Debora Lima Santos, Dilermando Queiroz Neto, Leonardo P Tizzei, Maria Julia De Castro Villafranca Garcia, Maysa Mal- fiza Garcia de Macedo, and Priscilla. Barreira Avegliano. Direct sampling for extreme weather generation. In Proceedings of the Annual Conference on Neural Information Processing Systems (NeurIPS), 2022.
Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio Lopez, and Vladlen Koltun. Carla: An open urban driving simulator. In Conference robot learning, pp. 1-16. PMLR, 2017.
Jessica Echterhoff, An Yan, Kyungtac Han, Amr Abdelraouf, Rohit Gupta, and Julian McAuley. Driving through the concept gridlock: Unraveling explainability bottlenecks in automated driving. In Proceedings of the IEFF/CVF Winter Conference on Applications of Computer Vision, pp. 7346 7346-7355, 2024.
28

---

Published in Transactions on Machine Learning Research (01/2026)
Zaid A El-Shair, Abdalmalek Abu-raddaha, Aaron Coficld, Hisham Alawneh, Mohamed Aladem, Yazan Hamzeh, and Samir A Rawashdeh. Sid: Stereo image dataset for autonomous driving in adverse conditions. In NAECON 2024-IEEE National Aerospace and Electronics Conference, pp. 403 408. LEEE, 2024.
Jeffrey L. Elman. Finding structure in time. Cognitive Science, 14(2):179 211, 1990. doi: 10.1207/
s15516709cog1402__1
Jianwu Fang, Lei-Lei Li, Kuan Yang, Zhedong Zheng, Jianru Xue, and Tat-Seng Chua. Cognitive accident prediction in driving scenes: A multimodality benchmark. arXiv preprint 2ii:2222.09981,, 2022.
Akshay Gopalkrishnan, Ross Greer, and Mohan Trivedi. Multi-frame, lightweight & efficient vision-language models for question answering in autonomous driving. ar arXio:2403. 19838, 2024.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), pp. 770-777 778, 2016.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8):1735 1780, 1997.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weishu Chen. Lora: Low-rank adaptation of large language models. arXie preprinl or orXiv:2106. 0xxx:2106.00685, 2021a.
Haofeng Hu, Liang Chen, Zhongyuan Wang, Lei Zhu, Mengyang Zhang, and Yun Fu. Depth-attentional features for single-image rain removal. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 8022-8031, 2019.
Xiaowei Hu, Lei Zhn, Tianyu Wang, Chi-Wing Fu, and Pheng- Heng. Single-image real-time rain removal based on depth-guided non-local features. IEEE Transactions on Image Processing, 30:1759 10:1759-1770, 2021b.
Jyh-Jing Hwang, Runsheng Xu, Hubert Lin, Wei-Chih Hung, Jingwei Ji, Kristy Choi, Di Huang, Tong He, Paul Covington, Benjamin Sapp, Yin Zhou, James Guo, Dragomir Anguelov, and Mingxing Tan. EMMA: End-to-end multimodal model for autonomous driving. Transactions on Machine Learning Research, 2025.
ISSN 2835-8856.
Jinkyu Kim, Anna Rohrbach, Trevor Darrell, John Canny, and Zeynep Akala. Textual explanations for self-driving vehicles. In Proceedings of the European conference on computer vision (ECCV), pp. 563-578, 2018.
Hari Chandana Kuchibhotla, Sai Srinivas Kancheti, Abbavaram Gowtham Reddy, and Vinceth N. Balasub- ramanian. Semantic alignment for prompt-tuning in vision language models. Transactions on Machine Learning Research, 2025. ISSN 2835-8856.
Zhengleng Lai, Haotian Zhang, Bowen Zhang, Wentao WU, Haoping Bai, Aleksei Timofeev, Xianzhi Du, Zhe Gan, Jiulong Shan, Chen-Nee Chuah, et al. Veclip: Improving clip training via visual-enriched captions. In European Conference on Computer Vision, pp. 111-127. Springer, 2024.
Boyi Li, Yue Wang, Jiageng Mao, Boris Ivanovic, Sushant Veer, Karen Leung, and Marco Pavone. Driving everywhere with large language model policy adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14948-14957, 2024a.
Changkang Li and Yalong Jiang. Vlavad: Vision-language models assisted unsupervised video anomaly detection. In 35th British Machine Vision Conference BMVC, Glasgow, UK. BMVA, 2024.
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. BLIP: Bootstrapping language-image pre-training for unified languuel understanding and generation. In Proceedings of the 39th International Conference on Machine Learning, volume 162, pp. 19888-12900. PMLR, 17-23 Jul 2022. URL https:/tt:/pprroeddia mlr. mlr. Attr.preas/v162//1122 html
29

---

Published in Transactions on Machine Learning Research (01/2026)
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning, pp. 19730 19742. PMLR, 2023.
Xiaofan Li, Yifu Zhang, and Xiaoqing Ye. Drivingdiffusion: Tayout-guided multi-view driving scenarios vidco generation with latent diffusion model. In European Conference Computer Vision, pp. 469-485. Springer, 2025.
Zizhao Li, Kourosh Khoshelham, and Joseph West. Contrastive class anchor learning for open set object recognition in driving scenes. Transactions Machine Learning Research, 2024b. ISSN 2835-8856.
Chin-Yew Lin. Rouge: A package for aulomatic evaluation of summaries. In Tex summarization branches out, pp. 74-81, 81, 2004.
Xin Lu, Jie Xiao, Yurui Zhu, and Xueyang Fu. Continuous adverse weather removal via degradation-aware distillation. In Proceedings of the Computer Vision and Pattern Recognition Conference (CVPR), pp. 28113 28123, June 2025.
Ge Ya Luo, ZhiHao Luo, Anthony Gosselin, Alexia Jolicocur-Martincan, and Christopher Pal. Ctrl-v: Higher fidelity autonomous vehicle video generation with bounding-box controlled object motion. Transactions on Machine Learning Research, 2025. ISSN 2835-8856.
Ji Ma, Wei Suo, Peng Wang, and Yanning Zhang. C31: Content correlated vision-langues instruction tuning data generation via contrastive learning. In Proceedings of the Thörty-Thin Third International Joint Conference on Artificial Intelligence, IJCAI-24, pp. 1155-1163, 2024a.
Yunsheng Ma, Can Cui, Xu Cao, Wenqian Ye, Peiran Liu, Juanwu Lu, Amr Abdelraouf, Rohit Gupta, Kyungtae Han, Aniket Bera, et al. Lampilot: An open benchmark dataset for autonomous driving with language model programs. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15141-15151, 2024b.
Adyasha Maharana, Amita Kamath, Christopher Clark, Mohit Bansal, and Aniruddha Kembhavi. Expos- ing and addressing cross-task inconsistency in unified vision-language models. Transactions on Machine Learning Reseorch, 2024. ISSN 2835-8856.
Srikanth Malla, Chiho Choi, Isht Dwivedi, Joon Hee Choi, and Jiachen Li. Drama: Joint risk localization and captioning in driving. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pp. 1043-1052, 2023.
Aboli Marathe, Deva. Ramanan, Rahee Walambe, and Ketan Kotecha. Wedge: A multi-weather autonomous driving dataset built from generative dision-language models. In Proceedings of the JEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, pp. 3318 3327,,377
June 2023.
Ana-Maria Marcu, Long Chen, Jan Hünermann, Alice Karnsund, Benoit Hanotte, Prajwal Chidananda, Saurabh Nair, Vijay Badrinarayanan, Alex Kendall, Jamie Shotton, et al. Lingoqa: Visual question answering for autonomous driving. In Europeon Conference on Computer Vision, pp. 252 269. Springer, 2024.
Lucas Moeller, Pascal Tilli, Thang Vu, and Sebastian Pad6. Explaining caption-image interactions in CLIP models with second-order attributions. Transactions Machine Learning Reseurch, 2025. ISSN 2835- 8856.
Valentina Musat, Ivan Fursa, Paul Newman, Fabio Cuzzolin, and Andrew Bradley, Multi-weather city: Ad- verse weather stacking for autonomous driving. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 2906 2915, 2021.
Chenbin Pan, Burhaneddin Yaman, Tommaso Nesti, Abhirup Mallik, Alessandro G Allievi, Senem Veli- pasalar, and Liu Ren. Vip: Vision language planning for autonomous driving. In Proceedings of the IEFF/CVP Conference on Computer Vision and Patlern Recognation, pp. 14760-14766 2024.
30

---

Published in Transactions on Machine Learning Research (01/2026)
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: method for automatic evaluation of machine translation. In Proceedings of the 40th, Annual Meeling of the Associalion for Compulational Linguistics, pp. 311 318, 2002.
SungYeon Park, MinJae Lee, JiHHykk Kang, Hahyeon Choi, Yoonah Park, Juhwan Cho, Adam Lee, and DongKyu Kim. Vlaad: Vision and language assistant for autonomous driving. In Proceedings of the IEEE/CVP Winter Conference on Applications of Compuler Vision, pp. 980- 987, 2024.
Tianwen Qian, Jingjing Chen, Linhai Zhuo, Yang Jiao, and Yu-Gang Jiang. Nuscenes-qa: A multi-modal visual question answering benchmark for autonomous driving scenario. In Proceedings of the AAAI Con- ference on Artificial Intelligence, volume 38, pp. 4542-4550, 2024.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramosh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference machine learning, pp. 8748-8763. PMLR, 2021.
Vasili Ramanishka, Yi-Ting Chen, Teruhisa Misu, and Kate Saenko. Toward driving scene understanding: A dataset for learning driver behavior and causal reasoning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognátion, pp. 7699-7707, 2018.
Christos Sakaridis, Dengxin Dai, and Luc Van Gool. Semantic foggy scene understanding with synthetic data. International Journal of Computer Vision, 126:973 992, 2018.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4510-4520. 2018.
Mustafa Shukor, Guillaume Couairon, and Matthieu Cord. Efficient vision-language pretraining with visual concepts and hierarchical alignment. In 33rd British Machine Vision Conference BMVC, London, UK. BMVA Press, 2022.
Chonghao Sima, Katrin Renz, Kashyap Chitta, Li Chen, Hanxue Zhang, Chengen Xie, Jens Beißwenger, Ping Luo, Andreas Geiger, and Hongyang Li. Drivelm: Driving with graph visual question answering. In European Conference on Computer Vision, pp. 256-27.. Springer, 2025.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition.
arXiv preprinl Xrx: 1556, 2014.
Tao Sun, Mattia Segu, Janis Postels, Yuxuan Wang, Lue Van Gool, Bernt Schiele, Federico Tombari, and Fisher Yu. Shift: a synthetic driving dataset for continuous multi-task domain adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Maric-Anne Lachaux, Timothée Lacroix, Bapliste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models. 3rXiv:2302.13971
Shihao Wang, Zhiding Yu, Xiaohui Jiang, Shiyi Lan, Min Shi, Nadine Chang, Jan Kautz, Ying Li, and Jose M Alvarez. Omnidrive: holistic vision-language dataset for autonomous driving with counterfactual reasoning. In Proceedings of the Computer Vision and Pallern Recognition Conference, pp. 22442 22452, 2025.
Yuqi Wang, Jiawei He, Lue Fan, Hongxin Li, Yuntao Chen, and Zhaoxiang Zhang. Driving into the future: Multiview visual forecasting and planning with world model for autonomous driving. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14749-14759, 2024.
Junjie Wu, Tsz Ting Chung, Kai Chen, and Dit- Yan Young. Unified triplet-level hallucination evaluation for large vision-language models. Transactions on Machine Learning Research, 2025. ISSN 2835-8856.
31

---

Published in Transactions on Machine Learning Research (01/2026)
Yiming Xie, Henglu Wei, Zhenyi Liu, Xiaoyu Wang, and Xiangyang Ji. Synfog: A photo-realistic synthetic fog dataset based on end-to-end imaging simulation for advancing real-world delogging in autonomous driving. In Proceedings of the LEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 21763 21772, June 2024.
Zhenhua Xu, Yujia Zhang, Enze Xic, Zhen Zhao, Yong Guo, Kwan- K Wong, Zhenguo Li, and Hengshuang Zhao. Drivegp14: Interpretable end-to-end aulonomous driving via large language model. IEEE Robolics and Automation Letters, 2024.
Fisher Yu, Haofeng Chen, Xin Wang, Wenqi Xian, Yingying Chen, Fangchen Liu, Vashisht Madhavan, and Trevor Darrell. Bdd100k: A diverse driving dataset for heterogeneous multitask learning. The International Journal of Robotics Research, 39(1): 18.
Yuan Zang, Tian Yun, Hao Tan, Trung Bui, and Chen Sun. Pro-trained vision-language models learn discoverable visual concepts. Transactions on Machine Learning Research, 2025. ISSN 2835-8856.
Jiawei Zhang, Chejian Xu, and Bo Li. Chatscene: Knowledgg-enabled safety-critical scenario generation for autonomous vehicles. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15499, 2024a.
Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 3836 3847, 2023a.
Renrui Zhang, Jiaming Han, Chris Liu, Aojun Zhou, Pan Lu, Yu Qiao, Hongsheng Li, and Peng Gao. Llama-adapter: Efficient fine-tuning of large language models with zero-initialized attention. In The Twelfth International Conference on Learning Representations, 2024b.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint ar Xiv:2200 01068, 2022.
Yue Zhang, Yajic Zou, Yunlong Zhang, Lingtao Wu, et al. Spatiotemporal interaction pattern recognition and risk evolution analysis during lane changes. IEEE Transactions Intelligent Transporlation Systems, 24(6):6663 6673, 2023b.
Chenhao Zheng, Jieyu Zhang, Aniruddha Kembhavi, and Ranjay Krishna. Tterated learning improves com- positionality in large vision-language models. Proceedings of the IEEE/CVP Conference on Computer Vision and Pattern Recognition, pp. 13785 13795, 2024.
32


Reviews:
[{"review_id": "x7hvrpBWA7", "reviewer": "Reviewer_EgsA", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper leverage VLM for autonomous driving scenes perception in transitional weather. To improve the performance of VLM in autonomous driving tasks, the authors integartes a feature encoder for transitional weather, and a lightweight backbone for robust visual feature extraction. A fusion module is proposed to balance visual and linguistic cues, as well as a text generator. New dataset is contributed and the evaluations are performed on this dataset.", "claims_and_evidence_yn": "Yes", "claims_explanation": "Extensive experiments are provided to support the authors claims", "audience_yn": "Yes", "audience_explanation": "It is important in VLM for autonomous driving", "requested_changes": "Please provide ablation of each proposed module in the pipeline, both qualitatively and quantitatively.", "broader_impact_concerns": "none", "additional_comments": "I'm not familiar with VLMs and its application in autonomous driving.", "extra_scores": {}, "extra_text": {}}, {"review_id": "aSgZEi1Md9", "reviewer": "Reviewer_g2A7", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper tackles weather-transition modeling in AVT-VQA. Its key contributions are:\n- SC-VAE, a data-synthesis framework that interpolates weather conditions from the Weather-Driving (WD) dataset.\n- VLAAD-TW, an efficient yet high-performing VQA architecture.", "claims_and_evidence_yn": "Yes", "claims_explanation": "**Strengths**\n\nOverall, the claims are well supported by sufficient experiements, e.g., \n1. The quality of the data are verified by Table 4 etc\n2. The validity of the model is verified by Table 5~9\n\n**Weaknesses**\n\n1.\tSignificance of transitions. If extreme conditions (e.g., cloudy and foggy) are already available, how much does modeling the intermediate transition help? An ablation that compares VLAAD-TW trained on (a) WD alone vs. (b) AIWD16-text is essential to quantify the gain.\n2.\tGeneralization of AIWD16-text. Demonstrate that models fine-tuned on the synthetic data do not degrade on realistic benchmarks; otherwise the utility of the new dataset is unclear.\n3.\tScope beyond perception. Because the target domain is autonomous driving, include QAs that go beyond scene understanding (e.g., decision-making, planning) to broaden impact.", "audience_yn": "Yes", "audience_explanation": "1.\tHighlights an under-explored data-augmentation angle (weather transitions) that deserves more attention.\n2.\tDelivers value on both the data (new benchmark) and model-design fronts.", "requested_changes": "Besides those mentioned above in the weaknesses：\n\n1.\tPSNR/SNR evaluation. These metrics need ground-truth frames. Are only the first (t = 0) and last (t = T) frames used, or are intermediate frames also available? Clarify protocol.\n2.\tTransition realism. Some sequences (rain → snow) look unnatural; please comment on physical plausibility or provide user-study validation.\n3.\tPre-training vs. fine-tuning. Specify which components of VLAAD-TW and the baselines (BLIP, etc.) are pre-trained, whether all models are fine-tuned on AIWD16-text, and whether DriveLM comparisons use identically tuned checkpoints. Missing details impede reproducibility.\n\nMinor\nTypo: “multi-model instruction tuning” → “multi-modal instruction tuning.” When a citation is used as a noun, drop the parentheses (e.g., “Park et al. 2024 introduced…”).", "broader_impact_concerns": "No", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "G2WTLeLaXm", "reviewer": "Reviewer_fY8v", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "To address these limitations that variable weather conditions for MLLM, this paper propose Vision-language Assistance for Autonomous Driving under Transitional Weather (VLAAD-TW). The VLAAD-TW integrates a Feature Encoder for Transitional Weather (FETW) for robust visual feature extraction, with a Spatiotemporal Contextual Aggregator (SCA), which models dynamic weather-induced changes. It also uses a Selective Attention-guided Fusion Module (SAFM) to balance visual and linguistic cues for a unified representation. Finally, a Semantic Text Generator (STG) fuses these representations to produce context-aware driving information. Furthermore, this paper introduce AIWD16-text dataset, an adverse intermediate weather driving dataset for vision language task.", "claims_and_evidence_yn": "Yes", "claims_explanation": "This paper provide experiments and dataset statistic to support the claims.", "audience_yn": "Yes", "audience_explanation": "The weather change may lead to a negative impact on vision language task for autonomous driving. And this paper provide a dataset and solution to address it.", "requested_changes": "(1) In fact, there are many robust benchmark, including weather changes for autonomous driving (e.g., Robo3D [1]). Author should clarify the difference except for providing caption, which is easy to obtain like OmniDrive [2].\n\n(2) I suggest authors providing more experiments about the VLAAD-TW trained on AIWD16-text. For example, first pretrain VLAAD-TW on AIWD16-text and then finetune it on nuScenes dataset and compare the performance with other methods (e.g., OmniDrive) to validate the effectiveness of VLAAD-TW and AIWD16-text.\n\n(3) This paper adopt generation models to produce different weather conditions, which is not reliable and may harm the performance of MLLM on the realistic data.\n\n[1] Robo3D: Towards Robust and Reliable 3D Perception against Corruptions\n\n[2] OmniDrive: A Holistic Vision-Language Dataset for Autonomous Driving with Counterfactual Reasoning", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)