TRIDE: A Text-assisted Radar-Image weather-aware fusion network for Depth Estimation
Authors: Huawei Sun
Zixu Wang
Hao Feng
Julius Ott
Lorenzo Servadei
Robert Wille
Venue: Accepted by TMLR ()

Abstract: Depth estimation, essential for autonomous driving, seeks to interpret the 3D environment surrounding vehicles. The development of radar sensors, known for their cost-efficiency and robustness, has spurred interest in radar-camera fusion-based solutions. However, existing algorithms fuse features from these modalities without accounting for weather conditions, despite radars being known to be more robust than cameras under adverse weather. Additionally, while Vision-Language models have seen rapid advancement, utilizing language descriptions alongside other modalities for depth estimation remains an open challenge. This paper first introduces a text-generation strategy along with feature extraction and fusion techniques that can assist monocular depth estimation pipelines, leading to improved accuracy across different algorithms on the KITTI dataset. Building on this, we propose TRIDE, a radar-camera fusion algorithm that enhances text feature extraction by incorporating radar point information. To address the impact of weather on sensor performance, we introduce a weather-aware fusion block that adaptively adjusts radar weighting based on current weather conditions. Our method, benchmarked on the nuScenes dataset, demonstrates performance gains over the state-of-the-art, achieving a 12.87% improvement in MAE and a 9.08% improvement in RMSE. Code: https://github.com/harborsarah/TRIDE

Published in Transactions on Machine Learning Research (08/2025)
TRIDE: A Text-assisted Radar-Image weather-aware fusion network for Depth Estimation
Huawei Sun
Technical University of Munich Infineon Technologies AG
Zixu Wang
zinu wanu.anngtttmmmmmm de
Technical University of Munich Infineon Technologies AG
Hao Feng
Joo.feng@lummcco de
Technical University of Munich
Julius Ott
julius. ott-Btum.dd de
Technical University of Munich Infineon Technologies AG
Lorenzo Servadei
lorenzo.serraddeittam.ccm
Technical University of Munich
Robert Wille
robert. wwllrullil@tmmm.
Technical University of Munich
Reviewed on OpenReview: https: openreview. net/ forum? id= ZMgHnuMfse
Abstract
Depth estimation, essential for autonomous driving, seeks to interpret the 3D environment surrounding vehicles. The development of radar sensors, known for their cost-efficiency and robustness, has spurred interest in radar-camera fusion-based solutions. However, existing algorithms fuse features from these modalities without accounting for weather conditions, despite radars being known to be more robust than cameras under adverse weather. Ad- ditionally, while Vision-Language models have seen rapid advancement, utilizing language descriptions alongside other modalities for depth estimation remains an open challenge. This paper first introduces text-generation strategy along with feature extraction and fu- sion techniques that can assist monocular depth estimation pipelinos, lcading to improved accuracy across different algorithuns on the KITTI dalaset. Building on this, we propose TRIDE, radar-camera fusion algorithm that enhances text feature extraction by incorpo- rating radar point information. To address the impact of weather sensor performance, introduce weather-aware fusion block that adaptively adjusts radar weighting based on current weather conditions. Our method, benchmarked on the nuScenes dataset, demon- strates performance gains over the state-of-the-art, achieving a 12.87% improvement in MAE and 9.08% improvement RMSE. Code: htaps://ggthhub..cc//aabbbrorrs
1 Introduction
Estimating a dense depth map for a given scene essential for 3D scene reconstruction, which can be achieved using either an RGB image alone or a combination of an RGB image and sparse depth map. With advancements in deep learning, learning- based monocular depth estimation methods Lee al. (2019):
Li et al. Shao al. 2024 Patil et. al. (2022) Agarwal & Arora (023] Liu et. al. (2023): Piccinelli et al.

---

Published in Transactions on Machine Learning Research (08/2025)
WHICH MODALITY SHOULD WE TRUST?
Image
Describe the frame in five parts. The first part descibe the image in general. The second to the fifth parts describe the objecs in the right, middle right, middle left, and left part of the image
Image Encoder
Decoder
Depth
Weather- seateer--wwre Fusion Block
image depicts an urban street scene on a rainy day In the left part,
Generate image caption of the current image
Text Encoder
Weather Prediction 9
Text
In the middle left part, In the middle right part,
*A group of cars parked in the rain under a bridge."
Finally, in the right part,
Radar z
seems didn't notice some objects! Fortunately, the redar points dol
Radar- Enrichment Block
Ah, it's raining. We should trust more on the Redar!
Radar
Radar Encoder
Channels
Figure 1: Which modality should we trust? Our proposed text generation approach generates detailed general and regional descriptions from the image. However, both image and language descriptions are susceptible to adverse weather conditions, limitation that can be addressed by the robustness of radar.
2024aa have significantly outperformed traditional approaches Saxena et al. 2005 2008:: ; Wang et al. (0155 in terms of accuracy. To address the inherent challenges of this ill-
(0024: Yuan et al. 2022); Zeng et al.
posed problem, many studies Yin et al. 22019 Qi et al. (2018); Liu et al. (2023); Lee et al. (201 incorporate geometric cues and scene priors to enhance depth accuracy. For instance, WorDepth Zeng el al. (2024a)
leverages language as a prior to enhance the depth estimation process. However, as it generates only a single caption per frame, the impact of incorporating language limited, particularly in complex outdoor scenarios.
In the realm of depth completion, LiDAR-camera depth completion Hu al. 2021 Cheng ct al. (2020) Tang et al. (2024): Wang et al. (2023b); Yan et al. (2024 2023) Park et al. (2020) has achieved high performance by effectively processing irregularly sampled sparse depth points from LiDAR sensors. Recently, radar-camera depth estimation has gained popularity as a potential alternative to LiDAR, as radar sensors are cost-effective and perform robustly in adverse weather conditions. However, radar point clouds are substantially sparser and noisier than LiDAR, posing additional challenges. Current research Nobis et al.] (2019); Sun el. al.] (2023 2024a); Long et al. (2021): (Singh et al. (2023); [Snn et al. (2024b] 2025c aa) has focused primarily on addressing the sparsity and noise issues in radar data, with less emphasis exploring more effective fusion strategies to capitalize on the strengths of different sensors. With fusion methods proposed in (Singh et al. (2023] Sun et al. (2024b), the model is still not able to adjust the weight between image and radar features under different weather conditions. Given that radar sensors are particularly robust under adverse weather conditions, incorporating weather information could allow the radar branch to be more heavily weighted in challenging environments.
camera
To address the abovementioned limitations, we begin by exploring how to fully leverage language information in estimating depth. We utilize the capabilities of rapidly advancing Multimodal Large Language Models (MLLMs) to generate detailed descriptions of each frame. Specifically, propose text-generation strategy that describes the image in two scopes: general overview and region-specifc details. During decoding, these lext features are integrated into the model through general and regional attention blocks at two slages, effectively enhancing the decoding process. To demonstrate the impact of integrating language information on depth estimation, we first conduct experiments on the KITTI benchmark Geiger et al.] (2012). By incorporating the text branch into several existing monocular depth estimation algorithms, our approach achieves superior performance compared to the original models.
2

---

Published in Transactions on Machine Learning Research (08/2025)
Frad
Decoder
Radar
Branch
(c) WaFB
General Attention
16 fdec
Regional Attention
Unet Decoder
Depth D
Channels
Encoder
DASPP
32
dee
(ddec
Fimg
Image Branch
Twra
Image Encoder
Image
GAP
Weather Prediction y
32
abp
gen
Regional / General Attention
Paragraph
Text Branch
Text
CLIP
Encoder
-leve!
REB
Teg
Cross Attention
Trey
Attention
Point
pMi
Cross
Encoder
pMR
Radar Points
pk
Fimg
Figure 2: Model Architecture. Our model consists of four parts. Radar, image, and text branches encode features individually and send them into the decoder, outputting the final depth map. The text branch also predicts the weather of the current frame to ensure extracting reasonable weather-aware feature for the fusion process.
Language descriptions generated from images are often impacted by adverse weather conditions. For in- stance, in night-time or rainy scenarios, images can become blurry or obscured by water drops, reducing the quality of the generated text descriptions. This leads to the question: Which modality should we trust estimate depth more accurately? Figure. [ illustrates the overall concept of our proposed method, showing that incorporating text descriptions into radar-camera system yield further improvements, radar sensors remain robust under such challenging conditions. In this context, we introduce TRIDE, multi-modal depth estimation framework that integrates information from radar, camera, and language to enhance robustness. To correct and refine the text information during feature extraction, we propose a Radar-Enrichment Block (REB) that incorporates region-specific 3D radar point information into the text leatures. Additionally, we introduce a Weather-aware Fusion Block (WaFB) that adaptively weights the radar branch based on weather information extracted from image and text, effectively guiding the fusion of radar and image features. Text features are integrated into the decoder in way that is consistent with the monocular depth estimation framework. We train and evaluate approach on the nuScenes Caesar et al. (2020) dataset, widely recognized benchmark for radar-camera-basse solutions, where it achieves state-of- the-art performance compared to existing radar-camera depth estimation algorithms. To summarize, our contributions are as follows:
We introduce a novel text branch, incorporating text-generation and feature extraction strategies, along with general and regional attention blocks to effectively integrate language information into image features. Integrating this text branch into existing monocular depth estimation algorithms improves depth estimation accuracy the KITTI dataset.
We propose TRIDE, multi-modal framework that surpasses existing radar-cament depth estimation algorithms on the nuScenes dataset.
To maximize the robustness advantages provided by the radar sensor, we first propose REB to enrich the regional text features with robust radar point features. Then, we design novel fusion block, WaFB, that leverages weather information during fusion.
To the best of our knowledge, this is the first work leverage detailed language descriptions to enhance radar-camera depth estimation while accounting for weather conditions during the fusion process.
3

---

Published in Transactions on Machine Learning Research (08/2025)
2 Related Work
Language Models for Autonomous Driving. Vision-Language Models (VLMs) Li et al. 2022
2028
Radford et al. [202]]; Oquab et al. (023); Zhu et al. (2023) are designed to learn rich vision- language cor- relations from diverse datasets, enabling zero-shot predictions in downstream tasks. Despite their potential, leveraging language as an auxiliary modality to enhance performance in Autonomous Driving (AD) remains an under-explored area. Some recent works have begun investigating this potential in different AD tasks, The system in Kim et al.] (2020) learns predict action responses by summarizing visual obsorvations natural language. Keysan el al. Keysan al. (2023) introduced text-based representations to improve tra- jectory prediction. In 3D localization, several studies Kolmet et al. (2022); Xia et al. (2024] Wang et al.) (2023a) focus on matching text descriptions with 3D point clouds. DRAMA Malla et al. (0023) aims to localize risk objects using free-form language descriptions.
For depth estimation, only a few researchers have explored the integration of language descriptions. WorDepth Zeng et al. (2024a) learned the distribution of 3D scenes from text captions, while RSA Zeng et al. (2024b) predicts scale from language, providing an explicit scaling constraint by transforming relative depth into metric scale. However, these approaches rely on single caption per image, often insufficient for de- scribing complex outdoor driving scenes. Thus, generating meaningful language descriptions and effectively extracting useful text information for complex scenarios is still challenging.
Madar-Cament Depth Estimation. Radar data offers valuable depth priors in adverse weather condi- tions, but radar-camera depth estimation presents significant challenges due to the inherent sparsity and noise in radar data. Earlier approaches addressed the sparsity issue by reprojecting radar points from mul- tiple sweeps onto the current frame Lin et al.
2020); Lo & Vandewalle) (2021); Long et al. (2021). Recently,
the focus has shifted toward leveraging single-scan radar data (Singh et al. (023)): Sun et al. (2024b); Li
et al. (2024a); Sun et al. (2025e): Li et al.
(022bb;; Sun et al. (2025b) for better real-time performance.
mitigate sparsity, several methods Li
2024a Singh et al. [2023 ; Sun et al. (2024)) employ two-stage
architectures that first estimate a quasi-dense depth map; followed by the final depth estimation in a second stage. While GET-UP Sun al.
(222cc) completes depth estimation within one stage, which incorporates point cloud upsampling an auxiliary task to avoid introducing extraneous noise in the densification pro- cess. Besides addressing noise and sparsity issues, effectively fusing different modalities is equally important. Most existing approaches Lin et al. (2020): Lo & Vandewalle (2021); Li et al. (2024b) perform information aggregation by simply concatenating radar and image features. However, this straightforward concatenation may lead to ineffective convolution over many zero activations due to the sparse nature of radar features. To address this, gated fusion was proposed in Singh et al. (2023) to more effectively leverage radar information. As enhancement, CaFNet [Sun et al. (2024b) introduces confidence-aware gated fusion, incorporating predicted radar confidence map to ignore regions with noisy radar data. Cross-attention is used in Li et al.) (2024a) to integrate radar information into the image fcatures, which helps to focus on critical information from each modality.
Despite these advancements, existing approaches do not consider weather information during fusion. More- over, how to effectively incorporate text information to enrich radar-camera Iusion-based algorithms remains open question.
3 Approach
This section begins by presenting our text-generation method, followed by an overview of the proposed TRIDE architecture. Next, we introduce the text encoding and decoder structures. Subscquently, we discuss the loss functions used for training TRIDE. Finally, we explain how text can assist in monocular depth estimation.
3.1 Text Generation
To fully leverage the power of text-based information, rather than using image captioning models, we generate a detailed description of the input image by using a specially designed prompt as input to the model MiniCPM-V Yao et al. (2024). The MLLM, denoted as M, takes an image Ting and a prompt p as inputs, and outputs five paragraphs: The first paragraph tgen provides an overview of the image, including details
4

---

Published in Transactions on Machine Learning Research (08/2025)
such the current weather conditions. The remaining four paragraphs provide region-specific descriptions. Here the image is divided into four regions: left (T.), middle-left (MT.), middle-right (MR), and right (R); with descriptions tleg MLL MR tll reg "eg Frgg each containing n()) sentences, respectively. The descriptions for each region contain information about the objects present and their estimated depth within the scenc. Thercfore, language descriptions can serve as priors, providing additional information to support depth estimatiion. Additionally, region-specific descriptions offer more detailed insights into the image, allowing for casier alignment and fusion with image features. The designed prompt and an example of the text description are shown in the supplementary material.
3.2 Model Architecture
Our radar-camera fusion model, illustrated in Figure 2 processes information through three main branches. The image branch uses a ResNet-34 encoder He et al. (2016) to process an input image Timg E # generating five multi-scale features (Singli=1, where H and W denote the image's height and width, respec- tively. For radar information, given a radar point cloud E RNXC, we project the radar points onto the image plane, forming radar projection channels, Frad E Hx Hx,xxx(C,-2) (C, Here, and C. represent the number of radar points and the features per point, respectively. These channels contain information such as depth, velocity, and Radar Cross Section (RCS), which are processed by a ResNet-15 encoder He et al. (2016) extract 2D radar features.
The text branch processes the extracted text and radar point cloud r to produce a general text feature T Tgen and four region-specific text features, reg: TML, reg and TR reg. Each text feature is vector of dimension C.. Moreover, a weather-aware feature Tea is generated by combining the pooled final image feature fing with Tgen, enhancing the model's capability to predict weather-related information.
In the decoding phase, we utilize a UNet Ronneberger et al. (2015) structure. Image and radar features are fused through the Weather-aware Fusion Block (WaFBB, which incorporates the weather feature Twea. Additionally, text features are integrated at two stages: Tgen is fused at scale of म. while the regional features apply regional attention with the feature at scale of 1. Further details are provided in Sec. 3.4
3.3
Text Encoding
The text branch processes the extracted text alongside the radar point cloud r E NR. where N represents the number of radar points in the current frame, and C, denotes the size of the radar point features. The goal is twofold. First, the text sometimes contains roughly estimated depth information generated by the MILM, which can be noisy and inaccurate. Integrating radar point information into the text features helps rectify this inaccuracy. Second, in adverse weather conditions, images can appear blurred, leading to even noisier information in the text. By incorporating robust radar point information, the text feature extraction process is better guided, improving aligmment with the correct spatial positions.
To encode the textual information, we first employ the text encoder from the pro-trained vision-language model CLIP Radford et al. (021)) to capture sentence-level features within each paragraph. For example, given a paragraph theg containing nt, sentences, CL.IP encodes the features as fl (/..,t..... with each feature vector J. E..
To manage the variation in sentence counts across paragraphs and extract a unified feature representation for cach paragraph, we introduce a paragraph-level encoder block, illustrated in Figure 3 This block employs Long Short-Term Memory (LSTM) layers Hochreiter (1997) to generate global feature representation for each paragraph, leveraging LSTM's ability to capture long-term dependencies. By sequentially processing the sentence features, the LSTM extracts a comprehensive feature representation that encapsulates the primary information of each paragraph. After passing through this encoder block, we obtain general text feature, Tgen; as well as four region-specifie text. features, freg = (&le JMg /MR }- Experiments in Sec. 4.4.3 demonstrate the effectiveness of this block.
r
Additionally, we introduce a Radar- Cedri-EErrichment Block (REB) to enhance the regional text features with corresponding radar point features, as depicted in Figure 4 The radar point features are initially extted
5

---

Published in Transactions on Machine Learning Research (08/2025)
(a) Paragraph level Encoder
(b)
Block
PMR
LSTM
LSTM
LSTM
free
pl
Frea
PML
Freg
pR
O
Q
v
f
L
Treg
Cross Attention
Cross Attention
Cross Attention Tree
Cross Attention
Tri
Trea
Treg
Figure 3: Paragraph-level text encoding block. For paragraph containing sentences, the block in- cludes n LSTM layers, which sequentially process the sentences and output a final feature represent- ing the paragraph's global information.
Figure 4: Radar-Enrich block. This block takes regional-speciiie lext features and the radar features as input. The radar point features are divided into four regions to align with the text features. Then, the regional point features are used to enrich the re- gional text features through cross-attention blocks.
by the PointNet Qi et al. (2017), producing a point feature representation P E RNCC Based on the spatial positions of radar points, the point cloud can be divided into four regions, corresponding to their 2D projections on the image plane. Thus, the point feature P can be subdivided into four regional features: pL, phLl, PMR, and PR, These regional point features enrich the corresponding region-specific text features flig. feg. fag. : and fiy. by cross-attention mechanism Vaswani (0017) resulting in the final region-specifi text features TL TML, TMR, and TR Notably, the general text feature Tgen remains unaltered throughout reg.
this process.
As discussed in Sec. 3.1 the general description includes information about weather conditions. In the nuScenes dataset Cacsar ct al. 2020)) scones are categorized as normal, rainy, or nighttime. However, during lext extraction, certain frames labeled as rainy are sometimes described as "cloudy" in the generated text. This discrepancy limits the weather prediction accuracy for the "rainy' category, which remains around 75% when using only the general text feature, Tgen- To improve accuracy, we combine the pooled final image feature fing from the image encoder with Tgen, forming more robust weather feature Twea, for the final weather prediction and downstream fusion tasks.
x Specifically, fing E R x three-dimensional tensor to a vector of length Cimg- Next, an adaptive pooling layer is applied to ensure the image feature matches the dimensions of Tgen- The pooled image feature is then combined with T Tgen through element-wise addition, producing the final weather feature, Twea. A Multi-layer Perceptron (MLP) is subsequently applied to Twea to predict the final weather classification ÿ.
x xumx is first processed by a Global Average Pooling (GAP) layer to reduce the
3.4 Decoder
The decoder takes as input the image features (fing 1 radar features (iea and the weather feature Twons producing the final predicted depth map D. At each decoding stage, the image, radar, and weather WaFB, illustrated in Figure 5 During the fusion process, Twea provides
features fused through supplementary weather information to adjust the weighting of the radar branch. Additionally, the text features are fused into the decoding features in two stages, by proposed general attention and the regional attention blocks. Details are as follows:
Wcather-aware Fusion Block: As depicted in Figure 5 we extend the gatod-fusion mothod Singh et al. (0033) by computing an additional weight informed by weather data. Specifically, Twee is concatenated with the radar feature and passed through convolutional layer followed by sigmoid activation to generate the weather-aware weight
== o @Coonv(Twea & frad)),
(1)
where a()) indicates sigmoid activation function and indicates concatenation. Additionally, the radar feature passes through two separate branches to generate radar weight a, and a projection 3:
a (Coov((Fraa)
(2)

---

Published in Transactions on Machine Learning Research (08/2025)
B = ReLU(Conv((raa))
(3)
Consequently, the final fused feature is computed + B fing-
To incorporate Tgen into the fused feature fãee at a scale of 1 32 employ bi- directional cross-attention block (General Attention), as illustrated in Figure The Cross-Attention (CA) block defined as follows:
Text. Features Fusion:
CA(Q, =oftmaxt QKT
Vdy VV,
(4)
where Q, K. V indicate query, key, and value, respectively, and dy is the dimensionality of the keys. In the GA block, ggen is used as the query while fdee serves as the key and value, generating the first attention feature. Reversely, the second attention feature obtained by setting få as the query and Tgen as the key and value. These two attention features are then concatenated and processed through convolutional layer to produce a residual feature, which is added back to Sa formulating the output feature SGA- The GA block can be formulated as follows:
fGA = Conv(CA(TT WQ: fàce WK, face Wv) @ CA(face We, Tgen WK. : Tgen Wv)) + face.
(5)
This resulting feature is subsoquently processod by an upsampling module, and Densely connected Atrous Spatial Pyramid Pooling (DASPP) Yang et al. (0018) module, which enhances contextual feature extraction, yielding Sdee-
Then, the region-specific text features are integrated into fdee Here, fdee is divided horizontally into four segments (left, middle-left, migdle-right, and right), each undergoing bi-directional attention with the cor- responding regional text. features Tle TML, TMR and reg- The resulting four features are concatenated horizontally, forming (fare)' and passed to the next decoding phase. The UNet-Decoder then takes the decoding feature (fâte) the intermediate image and radar features {fimgliz (frad 1i=1 and the weather
reg
feature Twea input, estimating the final depth D.
(c) Weather-aware Fusion Black
Corv
Sigmold
B
Image
Encoder
Encoder
Encoder
1
Encoder
General Attention
Twres
Conv
Frud
Trep
Conv
Sigmaid
Text
CLIP
Paragraph: level Encoder
DASPP / PPM
fing
Depth
Decoder
Decader
Decoder
Regional Attenttion
Decoder
Figure 5: Weather-aware Fusion Block.
Figure 6: Model structure of integrating text infor- mation into image-only methods for KITTI dataset.
3.5
Loss Functions
Our model utilizes two loss functions to accomplish the tasks of depth estimation and weather classification. For weather classification, we apply cross-entropy loss to the ground truth label y and the predicted label ý:
L.css = Σ y log St
(6)
For depth estimation, we follow the approach in Sun et al. (2025c), using both the dense depth map and the single-scan depth D, as supervision:
Depph
1
Σ ( D(x)) [[]]++ =En,
1
Σ -
(7)
[2al
Ω FEQ
where the loss is computed only over the set of pixels where D2 or D values are valid.
The final loss is then obtained by summing the two individual losses L Wp weLcls where up and we the weighting factors.

---

Published in Transactions on Machine Learning Research (08/2025)
3.6 Text-assisted Monocular Depth Estimation
First, generale lext for the KITTI dataset. using our proposed texl-generation rules. Without any radar enrichment, the text is processed directly by the frozen pre-trained CLIP text encoder, followed by our proposed paragraph-level encoder, yiclding five text features as described in Sec. [33] As shown in Figure G these features are integrated into the image features through general and regional attention blocks, leading to the final depth prediction.
4 Experiments
Table 1: Performance Comparison on the muScenes Official Test Set.
Dist
Method
Sersuze R
Model
Metrics
Paramns,
Puntimel
MAE
RMSE
AbaRte
log10
RMSE
dg
RC-PDA Long et ol.
2021
4.80M
0.0186
2.225
4.159
0.106
0.031
0.186
0.864
0.944
0.974
RC-PDA CC--PA-BHT Long
2021
2.50M
0.003
4.234
0.032
0.800
0.942
0.975
DORN Lo & Vaudewalle
2022
107 807.
0.020s
1.898
3.928
0.100
0.050
0.164
0.905
0.962
0.952
Radar.es
22.60M
0.378s
3.712
0.903
0.965
0.953
50ma
CaFNel Sun el al 2210
62 62.25M
0.1822
1.671
3.674
0.098
0.038
0.164
0.906
0.963
0.953
36.67M
0.137s
3.567
RadarCam. Denta
2024n
1,286
2,964
GET-UP Stan a..
30.76M
0.313s
2.857
0.913
0.977
0.988
THDDE (ORl
45.70M
0.031s
1.089
2.621
0.065
0.026
0.124
0.951
0.980
0.990
long, et
4.80M
0.01184
3.338
6.653
0.122
0.050
0.822
0.323
0.345
RC-PDA
Loog ot
2012
2.59M
0.008
3.514
7,070
0.127
0.032
0.235
0.812
0.914
0.960
DORN Lo
Vandewwlle
2012
107. SSM
0.020s
2.170
4.532
0.105
0.055
0.170
0.896
0.960
0..800
Nad
22 80M
0.378
2.073
4.591
0.105
0.043
0,181
01896
0.962
01981
70m
CaFNet
Sum 05 al.
62. 22.25M
0.132s
2,010
4.193
0.101
0.040
0.174
0.897
0.961
0.983
Li
202241
36.87M
0.137s
1.823
4.304
RodarCan: Camn-1
Deaiin 1.
al 21224
1.588
3.663
GET-UP Sun
30.76M
0.313s
1.541
3.657
0.075
0.032
0.145
0.938
0.974
0.986
TRime (OLLS
45.70M
0.031s
1.345
3.330
0.068
0.028
0.133
0.946
0.978
1966
0.989
BTS
(2019
33.65M
3.125
0,038
0.800
0.977
AdaBian
Braat et
2122
78. 15M
0.28-1s
3.511
5.865
0.197
0.089
0.261
0.612
0.612200
0.929
0..77
P3Depth
0.304s
3.130
5.538
0.165
0.065
0.222
0.804
0.934
0.934-00
0.974
Lephhphh
Soag ml
73.14M
0.2576
2.511
5.151
0.117
0.029
0.187
0.865
0.953
0.980
Yuar
270.
03455
4.314
0,100
0.042
0.910
0.967
0.985
S2D Ma & Karreaa
2111
11.40M
0.007'
2.374
3.628
0.115
0.876
0.949
0.974
RC- RCAP
Tour
4,80 4.80M
0.0184
3.721
7,6322
0.126
0.033
0.238
0.813
0.914
0.960
80m
RC-PDA-IIC
IIG Lovig ot ni.
21721
2.50M
0.000x
3.664
7.775
0.138
0.064
0.245
0.806
0.009
0.957
DORN
Vitadewwlll
107 3sM
0.0200
2.4322
3:304
0.107
0.036
0.177
0.800
0.940
0.981
Nedar.Ne
Stanel
22, 22.80M
0.378
2.179
4.599
0.106
0.041
0,184
0.891
0.959
01980
CaFNet
62-25M
0.1322
2.100
4.705
0.101
0.040
0.176
0.895
0.959
0.981
Li el
H
36,87M
0,137s
1.927
4.610
RodarCamn- Desta La
2022
1.890
3.948
GET-UP Sun
30.76M
0.313e
1.632
3.932
0.076
0.032
0,148
0.931
0.971
0.986
TRIDE (CRLES
45.70M
0.031s
1.422
3.575
0.068
0.028
0.135
0.944
0.977
0.968
Dist, I, and R denotes evaluation distance, image, and radar, respectively.
There no available code for testing on the nuScenes dataset. Thus, not able calculate the number of parameters and the runtine of the model.
This section starts by explaining the datasets and the details of the implementation. Then, we evaluate the quantitative and qualitative of the depth estimation task. Finally, we conduct several ablation studies to demonstrate the effectiveness of the proposed blocks further.
4.1 Datasets and Implementation Details
nuSeencs Datasct. To validate our TRIDE, we use the nuScencs dataset Cacsar et al. (2020), a com- prehensive multi-sensor dataset. This dataset includes 700, 150, and 150 scenes in its training, validation, and test subsets, respectively, covering diverse driving conditions such normal, rainy, and night-time scenarios. For our experiments, we utilize the front camera and single-scan radar inputs to train and assess the proposed algorithm's 's effectiveness.
As observed in prior work Li et al. (024b)), accumulating LiDAR frames from past and future instances can introduce significant errors, leading to inaccurate ground truth depth maps. We also found that accumulated dense depth maps struggle to accurately capture the shape of individual objects. Since the dataset lacks official dense ground truth depth maps, developing effective supervision is crucial. In this study, we generate
8

---

Published in Transactions on Machine Learning Research (08/2025)
Image
Ground
RadarNet
GET-UP
Normal
Night time
Rainy
Figure 7: Qualitative comparison on nuScenes test set. Column shows the RGB image; column 2 plots the ground truth depth map. We compared our results with those of the RadarNet and GET-UP at an 80-meter depth range.
dense depth map D as follows: we first train NewCRFs Yuan al. (2022) using the single-scan depth map D., which accurately captures object shapes and produces preliminary dense depth predictions. For each frame, we use NewCRI's to generate D and then refine it by inserting the existing values from D. into D, ensuring more accurate dense depth supervision. Both D and D, are used to supervise the training of our network. Further details are explained in the supplementary material.
Our model is implemented PyTorch Paszke al. (2019) and trainod on an Nvidia Tesla A30 GPU with batch size of 10. We the Adam optimizer Kingma & Ba (2017) with initial learning rate of le-1, adjusting it based polynomial decay schedule with a power of p 0.9. To prevent overfitting, apply data augmentation techniques, including random flips and adjustments to contrast, brightness, and color. Additionally, random cropping to 448 x 800 pixels is performed during training to further enhance model robustness.
KITTI Dataset. The KITTI dataset Geiger al. (012 is the most widely used benchmark for outdoor depth estimation in driving scenarios. We evaluate the efficacy of our proposed text branch by integrating it into existing depth estimation algorithms. The models retrained and evaluated using the KITTI Eigen split Eigen et al. (2014). with all implementation details consistent with the original publications.
4.2 Quantitative Results
Evaluation on nuScenes. In this study, we benchmark our proposed TRIDE against existing radar- camera depth cstimation mothods and camera-only algorithms using standard evaluation metrics. The models are lested on the official nuScenes test set al full image resolution across three distance ranges: up 50 meters, 70 meters, and 80 meters, as detailed in Table Compared to camera-only methods, incorporating radar data significantly improves depth estimation accuracy while requiring fewer model pa- rameters. Against existing radar-camera algorithmss, our TRIDE model achieves superior performance over Sun et al. (2025c), with improvements of 12.25%, 12.72%, and 12.87% in MAF, and 8.26%, 8.94%, and 9.08% in RMSE at evaluation distances of 50, 70, and 80 meters, respectively, underscoring its efficacy in precise depth estimation. Furthermore, we evaluate the computational efficiency of the proposed method: TRIDE comprises 45.70 million parameters, million fewer than the current state-of-the-art, and exhibits a faster runtime, further demonstrating the architectural advantages and efficiency of the proposed approach. Importantly, reported model size and runtime measurements refer solely to the depth estimation stage; lext generation and CLIP encoding are evaluated separately. Further details on runtime analysis are provided in the supplementary material.
Evaluation on KITTL. We train five monocular depth estimation algorithms by integrating the proposed text branch into the original models, as detailed Table 2 Standard evaluation metrics, described the supplementary material, are used for comparison. Integrating the text branch adds approximately million (M) parameters to each model. Nevertheless, in most cases, the models with the added text branch
9

---

Published in Transactions on Machine Learning Research (08/2025)
outperform the originals. For instance, in the case of NewCRFs, 1.6% increase in parameters results in 3.1% improvement in RMSE.
Table 2: Quantitative results on the KITTI dataset.
Model
Abs Rel $
Sq Rel 4
RMSE 4
Riog
81 1
02 1
BTS Loe et al
2019
0.060
0.219
2.798
0,096
0.955
0.993
BTS +
0.060
0.210
2.458
0.092
0.958
0.994
NewCREFs Yuan et al.
2022
0.052
0.155
2.129
0.079
0.974
0.997
CewC
0.051
0.149
2.064
0.077
0.975
0.996
DPT Ranftl et al.
2021
0.062
0.222
2.575
0.092
0.959
0.995
DP1
0.061
0.196
2.345
0.089
0.964
0.996
PixelFormer Agarwal Arora
2023
0.051
0.149
2.081
0.077
0.976
0.997
Former
0.150
2.074
0.079
0.976
DDDiss Shao
20244
0.142
2.011
0.075
0.978
0.998
IEBins
0.050
0.143
2.006
0.074
0.978
0.998
' T' indicates integrating text branch inside.
4.3
Qualitative Results
Here, we compare the depth maps generated by our TRIDE with those from existing algorithms Singh el al. 2023 : Sun et al. (2025c) at a depth range of 80 meters under various weather conditions. is evident that TRIDE captures more detailed information, such as the shape of the street sign and the concrete mixer in the first row. Additionally, our model demonstrates improved resilience stripo-wise artifacts at night compared to RadarNet and GET-UP. In rainy conditions, TRIDE accurately captures distant vehicles, including cars and trucks, as shown in the third row.
4.4 Ablation Studies
To further demonstrate the efficacy of proposed methods, conduct scries of ablation studies to verify the effectiveness of each component.
4.4.1 Feature Dimensions Analysis
This section conducts experiments to determine the optimal combination of text feature dimension C, and radar point feature dimension C₁, while also considering the model's parameter count. The results are presented in Table 3 Note that C = N/A indicates that the paragraph-level text features extracted without using the radar-enrichment block, further highlighting the efficiency of the proposed block. For optimal effectiveness and efficiency, we select feature dimensions of C. = 128 and C = 256.
Table 3: Ablation study on the feature dimension.
C,
C²
MAE 1
RMSE
Absite
51 f
Params
128
N/A
1.468
3.640
0.072
0.942
44.71M
128
64
1.430
3.582
0.070
0.944
45.45M
128
128
1.435
3.591
0.069
0.942
45.53M
128
256
1.422
3.575
0.068
0.944
45.70M
128
512
1.432
3.589
0.069
0.943
46.03M
256
N/A
1.471
3.660
0.072
0.942
51.83M
256
64
1.426
3.570
0.069
0.944
53.03M
256
128
1.437
3.571
0.070
0.943
53.18M
256
256
1.426
3.578
0.069
0.943
53.48M
256
512
1.438
3.588
0.070
0.943
54.07M
C, is the text branch dimension, C. is the radar branch dimension.
4.4.2 Modality Analysis
Here, we first analyze the impact of using different modalities on the depth estimation lask, selecting the optimal feature dimensions for the text and radar point features based on the findings from the previous section. The results are presented in Table 4 Our TRIDE model achieves a 30.09% improvement in MAE over the image-only model. Furthermore, incorporating the text branch enhances the imago-radar model by additional 8.20% in MAF.
10

---

Published in Transactions on Machine Learning Research (08/2025)
Subsequently, examine the most effective fusion stages for integrating text and decoding features, specifi- cally determining the optimal placement of the General Attention (GA) and Regional Attention (RA) blocks, presented in Table
Table 4: Ablation study on the modality analysis.
Table Ablation study on text fusion block place- ment.
Modalities
MAE
RMSE
AbeRtl
6,
Params
2.034
4.553
0.103
0.905
29.7M
GA
RA
MAE 4
RMSE 4
AbsRel
61
I+T
1.872
4.213
0.086
0.918
31.7M
I+R
1.549
3.767
0.073
0.936
40.4M
1/32
1/32
1.437
3,602
0.070
0.942
I+ +
1,122
3.611
0.070
0.910
44.71M
1/32
1/16
1.422
3.575
0.068
0.944
++R+T
1.422
3.575
0.068
0.944
45.7M
1/32
1/8
1.432
3.592
0.069
0.092
R, and represent image, radar, and text modalities. TT indi-
1/16
1/32
1.441
3.615
0.070
0.942
removing radar point encoder and REB from the text branch.
1/16
1/16
1,400
3.596
0.069
0.943
Table 5: Ablation study on text generation process.
1/16
1/8
1.434
3.601
0.070
0.943
1/32
w/o
1.499
3.660
0.070
0.942
Text Generation
MAE
RMSE 1
Absitel +
61
w/o
1/16
1.472
3.661
0.071
0.941
Captioning
1.531
3.801
0.074
0.937
w/o
w/o
1.476
3.632
0.071
0.942
Ours
1.422
3.575
0.068
0.944
'w/o' indicates removing the certain block from the final model.
4.4.3 Text Generation and Encoding Analysis
Here, we first evaluate the effectiveness of our proposed text generation process by comparing it to generating single image caption using the captioning model from Zeng et al. (2024)). as shown in Table 5 Since only one caption is generated per image, we encode it using the CLIP text encoder and fuse this text feature into the final model via the general altention block, omitting the regional at attetion block in this process. This paper also proposes a paragraph-level text encoding block to extract unified feature representation for each paragraph, loveraging the capability LSTM layers. Thus, we present additional experiments to evaluato the elfectiveness of the proposed block, presented in Table Specifically, compare our approach with alternative methods: averaging the text features within each paragraph (avg), concatenating the text features along the channel dimension followed by average pooling and convolutional layer (pool), using LSTM layers but averaging all the processed hidden features (LSTM_as) __ag), and more straightforward methods like wiement-wise addition, multiplication, and attention. The results indicate that simply averaging or applying pooling techniques fails to capture the global key information of the entire paragraph. Additionally, averaging the hidden states of the LSTM disrogards both the sequential information and the relative importance of features within the paragraph.
Table 7: Ablation study on the paragraph-level encoding block.
Text Encoding
MAE +
RMSE $
AbsRel
6 1
avg
1.544
3.838
0.074
0.936
pool
1.529
3.744
0.073
0.936
LSTM
1.422
3.575
0.068
0.944
LSTM avg
1.451
3.603
0.069
0.943
addition
1.469
3.651
0.070
0.941
multiplication
1.473
3.641
0.075
0.938
artention
1.437
3.579
0.070
0.944
4.4.4 Fusion Block Analysis
Table
compares the proposed WaFB with concalenation, elernent-wise addition, and gated fusion Singh
et al. 2023
Table 8: Ablation study on the fusion block.
Fusion
MAE
RMSE
AbsRtel
61 1
concatenation
1.572
3.901
0.075
0.931
addition
1.601
3.929
0.075
0.928
gated-fusion
1.526
3.737
0.072
0.936
WaFB
1.422
3.575
0.068
0.944

---

Published in Transactions on Machine Learning Research (08/2025)
4.5 Quantitative Results under Different Weather Conditions
In this work, we also evaluate the quantitative results under three weather conditions: normal weather, rainy, and night-time. Results under normal weather are presented in the appendix. The test set of nuSeenes Caesar jet al. (2020) is split into three subsets, containing 4254, 900, and 704 samples, respectively. In the following, we compare our method against RadarNet Siigh et al. (2023). GET-UP Sun al. (2025c), ; and our model, but with the gatod-fusion technique.
(a) Night-time Scenario
(b) Rainy Scenario
Metries
Metries
Dist
Method
Dist
Method
MAE
RMSE $
f
6 1
MAE
RMSE
6, 1
6g
RavlarNer GET-UP
Singh at al
2023
2.343
4.686
0.840
0.935
RadarNet
Singh et al.
2023
1.459
3.535
0.918
0.967
50m
Sun at al Ours [GF]
2025c
2.075
4.537
0.886
0.949
50m
GET-UP
Sum et a. A22c
1.018
2.605
0.954
0.979
2,069
0.953
(GE)
0.926
2.466
0.958
0.982
Ours
1.876
4.151
0.898
0.955
Ours
0.891
2.421
0.960
0.982
RadarrNet GET-UP
Singh at al
2023
2.565
5.937
0.830
0.928
RadarNet Singth et ml GET-UP
2023
1.672
4.116
0.903
0.966
70m
Sun et a. M22c
2.632
5.917
0.872
0.941
70m
Sum et a.
A2ac
1.204
3,162
0.950
0.977
Ours [GF]
2.582
5.631
0.947
(GE)
1.083
2.906
0.955
0.981
Ours
2.364
5.329
0.885
0.949
Ours
1.042
2.926
0.958
0.981
BadarNer
Singh et al.
2023
3.014
6.340
0.827
0.926
RadarNet
Singh et al.
2023
1.750
4.361
0.913
0.964
GET-UP
Sun et al 2025c
2.791
6.356
0.868
0.939
GET-UP
Sun et a. M22ac
1.275
3,392
0.949
0.977
80m
80m
Ours [GF]
2.725
6.001
0.806
0.945
(GF)
1.142
3.173
0.951
0.980
Ours
2.504
5.707
0.882
0.948
Ours
1.102
3.138
0.957
0.981
Table 9: Performance comparison under different weather and lighting scenarios.
The results demonstrate that, under rainy conditions, our TRIDE model outperforms the the-er GET-UP Sun et al.] (2025c) by 12.48%, 13.46%, and 13.53% in MAE at evaluation distances of 50, 70, and 80 meters, respectively. Additionally, it is evident that depth estimation algorithms perform the worst under night-time scenarios. However, compared to GET-UP, TRIDE achieves RMSE improvements of 8.51%, 9.94%, and 10.21% at 50, 70, and 80 meters, respectively. Furthermore, replacing gated fusion with our proposed WaFB significantly improves night-time performance, further demonstrating the effectiveness of the proposed fusion method.
Interestingly, when comparing GET-UP and RadarNet Singh et al. 2022) while GET-UP shows significant overall improvements over RadarNet on the full test set, it does not outperform RadarNet in RMSE under night-time scenarios and even performs worse at the 80-meter range. These findings highlight that night-time remains challenging case for depth cstimation. Nevertheless, TRIDE addresses this issue to some extent. demonstratinng its robustness in such adverse conditions.
5 Conclusion and Future Work
This paper introduces text generation and feature encoding strategy for depth estimation. To demonstrate the effectiveness of incorporating text information into image-only depth estimation models, we integrate text features into several monocular depth estimation algorithms and train them on the widely used KITTI dataset. The results indicate that adding text information enhances depth estimation accuracy. We further propose TRIDE, a radar-camera depth estimation algorithm trained and evaluated on the comprehensive nuScenes dataset. To leverage the robustness of radar sensors in adverse wcather conditions, we introduce radar-enrichment block that enhances lext features with radar point. features. Additionally, we propose a weather-aware fusion block that adaptively weights the decoding features from the radar and image branches based on weather conditions, fully utilizing radar's 's advantages in challenging environments. TRIDE surpasses the previous state-of-the-art model, achieving a 12.87% and 9.08% improvement in MAE and RMSE.
In this paper, we conduct a single MLLM to generate text descriptions by our proposed prompt, demonstrat- ing that incorporating text information improves depth estimation accuracy. In the future, a comparative study across multiple MLLMs will be explored to further assess how the robustness of generated text impacts depth estimation.
12

---

Published in Transactions on Machine Learning Research (08/2025)
6
Acknowledgement
Rescarch leading to these results has received funding from the EU ECSEL Joint Undertaking under grant agreement n" 101007326 (project AIACSM) and from the partner national funding authorities the German Ministry of Education and Research (BMBF).
References
Ashutosh Agarwal and Chetan Arora. Attention attention everywhere: Monocular depth prediction with skip attention. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 5861-5870, 2023.
Shariq Farooq Bhat, Ibraheem Alhashim, and Peter Wonka. Adabins: Depth estimation using adaptive bins. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4009-4018, 2021.
Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: A multimodal dataset for autonomous driving, In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pp. 11621-11631, 2020.
Xinjing Cheng, Peng Wang, Chenye Guan, and Ruigang Yang. Cspn++: Learning context and resource aware convolutional spatial propagation networks for depth completion. In Proceedings of the AAAI conference artificial intelligence, volume 34, pp. 10615 10622, 2020.
David Eigen, Christian Puhrsch, and Rob Fergus. Depth map prediction from single image using a multi- scale deep network. Advances in neural information processing systems, 27, 2014.
Andreas Geiger, Philip Lenz, and Raquel Urtasun. Are we ready for autonomous driving? the kitti vision benchmark suite. In Conference on Computer Vision and Pattern Recognition (CVPR), 2012.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and paltern recognilion, pp.
2016.
S Hochreiter. Long short-term memory. Neurol Computation MIT- MIT-Press, 1997.
Jia Cheng Hu, Roberto Cavicchioli, and Alessandro Capotondi. Expansionnet v2: Block static expansion in fast end to end training for image captioning. arXie preprint arXiv:2208.06:: 2022.
Mu Hu, Shuling Wang, Bin Li, Shiyu Ning, Li Fan, and Xiaojin Gong. Penet: Towards precise and efficient
image guided depth completion. In 2021 IEEE International Conference Robotics and Automation (ICRA), pp. 13656-13662. IEEE, 2021.
Ali Keysan, Andreas Look, Eitan Kosman, Gonca Gürsun, Jörg Wagner, Yu Yao, and Barbara Rakitsch. Can you text. what is happening? integrating pre-trained language encoders into trajectory prediction models for autonomous driving. arXiv preprint orXiv:2309. 05282, 2023.
Jinkyu Kim, Suhong Moon, Anna Rohrbach, Trevor Darrell, and John Canny. Advisable learning for self- driving vehicles by internalizing observation-to--ction rules. Proceedings of the TEFE Conference on Computer Vision and Pattern Recognition, pp. 9661 9670, 2020.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization, 2017.
Manuel Kolmet, Qunjie Zhou, Aljoša OSep, and Laura Leal-Taixé. Text2pos: Text-to-point--loud cross-modal localization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6687-6696, 2022.
Jin Han Lee, Myung-Ky Han, Dong Wook Ko, and Il Hong Suh. From big small: Multi-scale local planar guidance for monocular depth estimation. XrXio preprint arxie: 1rXi::1907.110 10326, 2019.
13

---

Published in Transactions on Machine Learning Research (08/2025)
Han Li, Yukai Ma, Yaqing Gu, Kewei Hu, Yong Liu, and Xingxing Zuo. Radarcam-depth: Radar-camera fusion for depth estimation with learned metric scale, 2024a. URL httpp:://arriiii http::///xxvvv...aaaaaae www.:oggg//bao..eooooooooooo.gso https///40000.00
Huadong Li, Minhao Jing, Jiajun Liang, Haoqiang Fan, and Renhe Ji. Sparse beats dense:
Rethinking supervision in radar-camera depth completion, 2024b. URL http:://aarrvv.orr//a///1111111
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In International conference on machine learning, pp. 12888 12900. PMLR, 2022.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning. pp. 19730-19742. PMLR, 2023.
Z Li, X Wang, X Liu, and Jiang. Binsformer: Revisiting adaptive bins for monocular depth estimation. arxiv 2022. arXiv preprint arXiv:224400098
Juan-Ting Lin, Dengxin Dai, and Luc Van Gool. Depth estimation from monocular images and sparse radar data. In 2020 IFFE/RSJ International Conference. on Intelligent Robots and Systems (TROS), pp. 10233 10240. IEEE, 2020.
Ce Liu, Suryansh Kumar, Shuhang Gu, Radu Timofte, and Luc Van Gool. Va-depthnet: A variational approach to single image depth prediction. ar preprint xXXi:2332.06556 2023.
Chen-Chou To and Patrick Vandewalle. Depth estimation from monocular images and sparse radar using deep ordinal regression network. In 2021 IEEE International Conference on Image Processing (ICIP), pp. 3343-3347. IEEE, 2021.
Yunfei Long, Daniel Morris, Xiaoming Liu, Marcos Castro, Punarjay Chakravarty, and Praveen Narayanan. Radar-camera pixel depth association for depth completion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12507-12516, 2021.
Fangchang Ma and Sertac Karaman. Sparse-to-dense: Depth prediction from sparse depth samples and single image. In 2018 TEFE international conference on robotics and automation (ICRA), pp. 4796-4803. IEEE, 2018.
Srikanth Malla, Chiho Choi, Isht Dwivedi, Joon Hee Choi, and Jiachen Li. Drama: Joint risk localization and captioning in driving. In Proceedings of the IEEE/CVF winler conference on applicalions of compuler vision, pp. 1043 1052, 2023.
Felix Nobis, Maximilian Geisslinger, Markus Weber, Johannes Betz, and Markus Lienkamp. deep learning- based radar and camera sensor fusion architecture for object detection. In 2019 Sensor Data Fusion: Trends, Solutions, Applications (SDF), pp. 1-7. IEEE, 2019.
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaacldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. Xiv preprint ar arXiv:2304.07111 2023.
Jinsun Park, Kyungdon Joo, Zhe Hu, Chi-Kuei Liu, and In So Kweon. Non-local spatial propagation network for depth completion. In Computer Dision-ECCH 2020: 16th European Conference, Glasgow, UK August 23 28, 2020, Proceedings, Port XIII 16, pp. 120 136. Springer, 2020.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang, Zach DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjic Bai, and Soumith Chintala. Pytorch: An imperative style, perfomarformance deep learning library. ArXiv, abs/1912.01703,, 2019.
14

---

Published in Transactions on Machine Learning Research (08/2025)
Vaishakh Patil, Christos Sakaridis, Alexander Liniger, and Lue Van Gool. P3depth: Monocular depth estimation with piecewise planarity prior. In Proceedings of the TEEE/CVF Conference on Compuler Vision and Pattern Recognition, pp. 1610 1621, 2022.
Luigi Piccinelli, Yung-Hsu Yang, Christos Sakaridis, Mattia Segu, Siyuan Li, Luc Van Gool, and Fisher Yu. Unidepth: Universal monocular metric depth estimation. In Proceedings of the IEEE/CVP Conference Compater Vision and Pallern Recognilion, pp. 10106-1000 10116, 2024.
Charles R Qi, Hao Su, Kaichun Mo, and Leonidas Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In Proceedings of the IEEE conference on compuler vision and pallern recognition, pp. 662- 2017.
Xiaojuan Qi, Renjie Liao, Zhengzhe Liu, Raquel Urtasun, and Jiaya Jia. Geonet: Geometric neural network for joint depth and surface normal estimation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 283-291, 2018.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. $748-8766 PML., 2021.
René Ranftl, Alexey Bochkovskiy, and Vladlen Koltun. Vision transformers for dense prediction. In Pro- ceedings of the IEEE/CVE international conference on computer vision, pp. 12179-12188, 2021.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical image compuling and compuler-assisled national conference, Munich, Germany, October 5-9, 2015, proceedings, part 111 18, pp. 2015.
-MCCCA 2015: 18th inter-
Springer,
Ashutosh Saxena, Sung Chung, and Andrew Ng. Learning depth from single monocular images. Advances in neural information processing systems, 18, 2005.
Ashutosh Saxena, Min Sun, and Andrew Y Ng. Make3d: Learning 3d scene structure from single still image. IEEE transactions on pottern analysis and machine intelligence, 31 840,820,-0000- 2008.
Shuwei Shao, Zhongcai Pei, Xingming Wu, Zhong Liu, Weihai Chen, and Zhengguo Li. lebins: Iterative elastic bins for monocular depth estimation. Advances in Neural Information Processing Systems, 36, 2024.
Akash Deep Singh, Yunhao Ba, Ankur Sarker, Howard Zhang, Achuta Kadambi, Stefano Soatto, Mani Srivastava, and Alex Wong. Depth estimation from camera image and mmwave radar point cloud. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 2023.
Minsoo Song, Seokjae Lim, and Wonjun Kim. Monocular depth estimation using laplacian pyramid-based depth residuals. IEEE Transactions on Circuits and Systems for Video Technology. 31(11):4381-438 2021. doi: 10.1109/TTSS
Huawei Sun, Hao Feng, Georg Stettinger, Lorenzo Servadei, and Robert Wille. Multi-task cross-modality allention-fusion for 2d object detection. In 2023 IEEE 26th International Conference on Intelligent Trans- portation Systems (ITSC), pp.
2023. doi: 10.1009/ISCC5777..222... 10421800.
Huawei Sun, Hao Feng, Gianfranco Mauro, Julius Ott, Georg Stettinger, Lorenzo Servadei, and Robert Wille, Enhanced radar perception via multi-task learning: Towards refined data for sensor fusion applications. In 2024 IEEE Intelligent Vehicles Symposium (IV), pp. 3179 3184, 2024a. doi: 10.1109/1IV55156.2222
10588795
Huawei Sun, Hao Feng, Julius Ott, Lorenzo Servadei, and Robert Wille. Cafnet: A confidence-driven framework for radar camera. depth estimation. arXio preprint or urXi::2407.00697 2024b.
15

---

Published in Transactions on Machine Learning Research (08/2025)
Huawei Sun, Bora Kunter Sahin, Georg Stettinger, Maximilian Bernhard, Matthias Schubert, and Robert Wille. Caraffusion: Improving 2d semantic segmentation with camera-radar point cloud fusion and sero- shot image inpainting. IEEE Robotics and Automation Letters, 10(7):7007 (07), 2025a. doi: 10.1109/
202.2225.3.72418.
Huawei Sun, Nastassia Vysotskaya, Tobias Sukianto, Hao Feng, Julius Ott, Xiangyuan Peng, Lorenzo Ser- vadei, and Robert Wille. Lircdepth: Lightweight radar-camera depth estimation via knowledge distillation and uncertainty guidance. In ICASSP 2025 2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 5, 2025b. doi:
Huawci Sun, Zixu Wang, Hao Feng, Julius Ott, Lorenzo Servadei, and Robert Wille. Gct-up: Geometric- aware depth estimation with radar points upsampling. In 2025 IEEE/CVF Winter Conference on Appli- cations of Computer Vision (WACV), pp. 1850-1860, 2025c. doi: 10.1109 /t////WAVV11..... 20210041..22501188...
Jie Tang, Fei-Peng Tian, Boshi An, Jian Li, and Ping Tan. Bilateral propagation network for depth comple- tion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1763-9772, June 2024.
A Vaswani. Attention is all you need. Advances in Neural Information Processing Systems, 2017.
Guangzhi Wang, Hehe Fan, and Mohan Kankanhalli. Text point cloud localization with transformer. In Proceedings of the AAAT Conference on Artificial Intelligence, volume 37, pp. 2023a.
Xiaoyan Wang, Chumping Hou, Liangzhou Pu, and Yonghong Hou. A depth estimating method from a single image using foe crf. Multimedia Tools and Applications, 74:9491-95000 9506, 2015.
Yufei Wang, Bo Li, Ge Zhang, Qi Liu, Tao Gao, and Yuchao Dai. Lrru: Loug- range recurrent updating networks for depth completion. In Proceedings of the IEFE/CV /CF international conference on computer vision, pp. 9422 9432, 2023b.
Yan Xia, Letian Shi, Zifeng Ding, Joao F Henriques, and Daniel Cremers. Text2loc: 3d point cloud lo- calization from natural language. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 14958-14967, 2024.
Zhiqiang Yan, Xiang Ti, Zhenyu Zhang, Jun Li, and Jian Yang. Rignet++: efficient repetitive image guided network for depth completion. arXiv preprint ar arXiv:2309.000 00050, 2023.
Zhiqiang Yan, Yuankai Lin, Kun Wang, Yupeng Zheng, Yufei Wang, Zhenyu Zhang, Jun Li, and Jian Yang. Tri-perspective view decomposition for geometry-aware depth completion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition, pp. 4874 4884, 2024.
Maoke Yang, Kun Yu, Chi Zhang, Zhiwei Li, and Kuiyuan Yang. Denseaspp for semantic segmentation in street scenes. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognation (CVPR), June 2018.
Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, Qianyu Chen, Huarong Zhou, Zhensheng Zou, Haoye Zhang, Shengding Hu, Zhi Zheng, Jie Zhou, Jie Cai, Xu Han, Guoyang Zeng, Dahai Li, Zhiyuan Liu, and Maosong Sun. Minicpm-v: A gpt-4v level mllm on your phone, 2024. URL https: http///axxvv..rree// http://aa/2400.008000
Wei Yin, Yifan Liu, Chunhua Shen, and Youliang Yan. Enforcing geometric coustraints of virtual normal for depth prediction. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 5684 5693, 2019.
Weihao Yuan, Xiaodong Gu, Zuozhuo Dai, Siyu Zhu, and Ping Tan. Neural window fully-connected erfs for monocular depth estimation. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pp. 3916 3916-3925, 2022.
16

---

Published in Transactions on Machine Learning Research (08/2025)
Ziyao Zeng, Daniel Wang, Fengyu Yang, Hyoungscob Park, Stefano Soatto, Dong Lao, and Alex Wong. Wordepth: Variational language prior for monocular depth estimation. In Proceedings of the IEEE/CVP Conference on Computer Vision and Pattern Recognition, pp. 9708 9719, 2024a.
Ziyao Zeng, Yangchao Wu, Hyoungseob Park, Daniel Wang, Fengyu Yang, Stefano Soatto, Dong Lao, Byung- Woo Hong, and Alex Wong. Rsa: Resolving scale ambiguitics in monocular depth estimators through language descriptions. arxiv preprinl or 10. 0..02924. 2024b.
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision- language understanding with advanced large language models. Xiv preprint arXiv:2304. 10592, 2023.
17

---

Published in Transactions on Machine Learning Research (08/2025)
A
Appendix
The supplementary material first analyzes the text descriptions, including the designed prompts and a comparison between our text descriptions and image captions, in See A.T Next, we examine our generated dense depth maps used for training supervision in Sec A.2 Following this, we detail the evaluation metrics in Sec A.3 Finally, we present comprchensive comparison of quantitative and qualitative results under different weather conditions in Sec A.4 and Sec A.6 and conclude in See A.10
A.1
Text Description
A.1.1
Prompt Design
In this work, we design a tailored prompt to generate detailed descriptions of a given image. These language- based descriptions serve as prior knowledge of the current scenario, aiding in the understanding of the image. The prompt is formulated as follows:
The Designed Prompt.
Describe the frame in five parts, and each part starts with dash sign (-). The first part describes the image in general, including the weather conditions.
The second to fifth parts describe the objects and estimate their depth (maximum 80 meters) in the right part, middle right part, middle left part, and left part of the image, respectively.
A.1.2 Visulization
In this section, we present a visualization of the lext description for given image. As shown in Figure S the description consists of five paragraphs. The first paragraph provides an overview of the image, while the second through fifth paragraphs describe the objects present and provide a rough estimate of their depth, progressing from the left to the right of the image. According to the lext description, it easy to align the regional text feature with the image features of the specific regions.
The emage disic a street wore cake cvarrnt wouline conditen,, eitly difinoo zu iggt stow and the city appiis to tw Biis with ciuite.
part, ic white
mauci road indicating nome
rekoaisd bull sciiehy non ts: The book is purtas anrtudion the aida of the roed, and pou bon
sesud og atrsde and contr buring to the siwwwyy the weviron move podeaation
rooing ta marked by while squeneu pain
sschail, loading up
50
inaye appeor in be angrniinamay 20 meteros away., glvns their retrive din caned
brcck and other oéjects The graasy embertinn Amennt behand seams
wawer thon Yusk and the trom. "hey ore roughty 16 maters away. comaidering pramiherce the trums. pedesttinn sigral indicates that
20
of more vegulae culliin and soblyyyy rosdend ere sear about matore away, julging by they bland
diulanzo These
Figure 8: Test description for a given image. The first paragraph describes the image in general way. Afterwards, the second to the fifth paragraphs introduce the image from left to right.
A.1.3 Text Description Comparison
In this work, also generate image captions using ExpansionNet v2 Hu et al. (2022), which is used in Zeng ct al. (2024a]b) b), for the given image and compare the generated descriptions under three different weather conditions.
Normal Weather Scenario. For the image presented in Figure g the generated caption is " "A truck driving down street with a traffic light". The descriptions generated using our proposed prompt are follows:
18

---

Published in Transactions on Machine Learning Research (08/2025)
Our Text Descriptions.
The image depicts street scene under overcast weather conditions, with diffused natural light suggesting either early morning or late afternoon. There no direct sunlight visible, and the sky appears to be filled with clouds.
the left part, there is a white tanker truck on the road, likely indicating some industrial activity nearby. The truck is parked on the side of the road, and its position suggests it may have stopped this point in time. 'Trees are prominent in this area, providing shade and contributing to the greenery of the environment. A podestrian crossing is marked by white squares painted the asphalt, leading up to the traffic lights.
Estimating from the perspective provided, the trees in the middle-left part of the image appear to be approximately 20 meters away, given their relative size compared to the truck and other objects. The grassy embankment behind them seems to be around 30 meters distant, based the slope and depth perception.
Moving towards the middle-right part, the traffic lights stand out as they are closer to the viewer than the truck and the trees. They are roughly 15 meters away, considering their prominence in the frame. The pedestrian signal indicates that pedestrians should not cross at this moment.
Finally, in the right part, beyond the immediate vicinity of the traffic signals, there's a hint of more vegetation and possibly residential buildings the far distance. These structures seem be about 60 meters away, judging by how they blend into the horizon line against the cloudy sky.
In this simple scenario, the caption provides general summary of the image. However, language- descriptions offer more detailed and contextually relevant information compared to the caption, enriching the understanding of the scene.
40
20
Figure 9: Example of rainy scenario.
Rainy Scenario. Figure 9 illustrates a rainy scenario. The generated caption is:
Image Caption.
A group of cars parked in the rain under bridge.
And our generated detailed descriptions follows:
19

---

Published in Transactions on Machine Learning Research (08/2025)
Our Text Descriptions.
The image depicts an urban street scene on rainy day. Visibility somewhat reduced due to the weather, and the wet road reflects the surrounding buildings and vehicles, adding a sense of depth the scene. The overcast sky suggests that it might be early morning late afternoon.
the left part, there are several cars parked along the side of street, with varying distances from the viewpoint. A dark-colored minivan is closest the camera, while further away, white and silver sedans can be seen parked in a line. The depth estimation for these objects ranges from approximately 10 meters (minivan) to about 50 meters (furthest sedan).
the middle left part, more details become visible as we move further down the street. There's a pedestrian walkway overhead connocting two buildings, which adds architectural interest and complexity to the scene. The cars continue to recede into the distance, with their depths estimated at around 30 to 40 meters from the viewer.
In the middle right part, the focus shifts slightly to show additional elements like a white delivery van and some construction barriers indicating ongoing work. These objects appear closer, with the van being roughly 20 to 30 meters away from the viewpoint, and the barriers appearing even nearer, possibly within 10 20 meters.
Finally, in the right part, the scene continues with the delivery van and construction activity, suggesting bustling area despite the inclement weather. Other parked vehicles and the edge of the street are visible, with the van likely being the closest object about 10 to 15 meters from the camera, and other elements fading into the background at varying distances up 80 meters.
In more complex scenarios, the captioning model provides only a high-level, generalized description of the image, which is often insufficient for accurate depth estimation. By contrast, our language descriptions offer more detailed and useful information, though they not without limitations. While the MLLM can effectively identify objects in the image, the prior depth values it predicts often contain significant errors. This underscores the critical role of our radar-enrichment block in correcting and refining depth predictions.
Night-time Scenario. Figure 10 illustrates a complex night-time scenario. We compare our text descrip- tion with the image caption of the given image. From the descriptions, it is evident that the image caption is entirely inaccurate, as there is no train the road. The captioning model struggles to handle such complex scenes, and incorporating this incorrect information into the depth estimation model could lead to confusion and degraded performance. In contrast, our text descriptions accurately identify that the bus is moving forward and barrier arm is present on the right, providing reliable and contextually correct information.
Image Caption. A road with train on the side of it.
And our generated detailed descriptions as follows:
20

---

Published in Transactions on Machine Learning Research (08/2025)
Our Text Descriptions.
The image depicts an urban setting during what appears to be twilight, with the sky displaying hues of blue and hint of pink near the horizon. Street lamps are on, providing illumination natural light fades. A bus is parked at what scems like toll plaza checkpoint.
the left part, there's railing along the edge of the road, leading up to a higher level. The railing suggests that this area might be elevated or bridge crossing. The street lamp in the middle-left part stands tall, casting its light over the scene. Its position indicates it could be approximately 10 meters from the viewpoint.
In the middle left part, beyond the railing, we can see gentle slope descending towards the road. This slope likely leads down another section of the roadway or parking lot. There's clear delineation between the two levels, suggesting some depth of around 20 meters.
The middle right part features the orange bus prominently. It occupies about half of the frame, indicating its size relative to the surroundings. The bus is positioned such that it were moving forward, it would travel straight ahoad into the distance. Judging by placement and perspective, it appears to be roughly 30 meters away from the viewpoint.
In the right part, besides the bus, there's a barrier arm partially extended, possibly controlling access to the area. Next to it is a yellow box with signage, which looks like payment kiosk or information booth. These objects suggest controlled entry point, perhaps for paid parking area restricted zone. Both the barrier arm and the box seem to be within 40 meters of the viewpoint.
70
50
20
Figure 10: Example of night-time scenario.
A.2 nuScenes Training Ground Truth Generation
In this section, compare different depth maps used for supervision. As shown in Figure I1 evaluate the single-scan depth map, the interpolated dense depth map generated using the method from Singh et al. )222), and our dense depth map produced by training NewCRFs Yuan et al. (2022).
The NewCRE' approach leverages traditional computer vision techniques, incorporating conditional random field blocks exploit obscrvation cues such as texture and positional information. This allows more effectively detect object contours. From the figure, it is evident that the interpolated depth map quite noisy compared to the CewS depth maps, with poorly defined object contours. These issues arise from
21

---

Published in Transactions on Machine Learning Research (08/2025)
compensation errors when mapping frames from the past and future to the current frame. However, also observed that the dense depth maps generated by NewCRFs contain errors. To address this, we integrate the accurate depth values from the single-scan depth map into the PewCRes-peedicted depth map, resulting in a refined dense depth map with reduced noise and improved accuracy.
Single-Scan Depth
Interpolated Depth
NewCRFs Depth
depthim)
depthtm)
depth (m)
Figure 11: Depth maps comparison. The first column visualizes the single-scan depth map generated by the LiDAR point cloud. The second column visualizes the interpolated depth maps utilizing the method from Singh et al. (2023), and the last column visualizes the dense depth map we use for the supervision.
A.3 Evaluation Metrics
Table 10 outlines the ovaluation metrics employed in this study for comparing performance. Here, denotes the set of 2D pixels for which ground truth LiDAR depth values are available.
Table 10: Metries definition for depth estimation task.
Definition
MAE
Σ men Id(x)) dgt (x)
RMSE
Exen (d(x) - dg 22)2/2
[2
AbsRel
Exen (d(x) d (z)) dg(((r)
log10
Ezen ll8110 d(x) - log10 dgt (x)
RMSFlog
Σ logio d(x:) log (x)]12
8, Kd(x)) maal d(z) dg((x)
6, Thre
[x((a) d()) < 1/21"
A.4 Quantitative Results under Different Weather Conditions
In this section, provide the comprehensive results under different weather conditions, presented in Table Table 12 and 13
22

---

Published in Transactions on Machine Learning Research (08/2025)
Table 11: Performance Comparison on sub Test Set under Normal Weather.
Metrics
Eval Distance
Method
MAK
RMSE
Abeltel 4
RMSElogg
RadarNel
Singh et al.
2023
1.653
3.630
0.096
0.159
0.910
0,970
50m
GET-UP
Sun et al
2725c
1.150
2.632
0.065
0.122
0,950
0.982
Ours [GF]
1.078
2.540
0.061
0.116
0.962
0.983
Ours
1.001
2.410
0.059
0.113
0.958
0.984
RadarNet Singh et ml.
2023
1.978
4.411
0.098
0.169
0.903
0.966
70m
GET-UP
Sun et al.
A125c
1.432
3.387
0.068
0.131
0.944
0.979
Ours [GFT
3.255
Ours
1.240
3.084
0.121
0.953
0.982
RadarNet Singh 2023
2.075
4.690
0.172
0.902
0.965
GET-UP
al 2725c
1.516
3.645
0.069
0.134
0.912
0.979
80m
Ours [GF)
1.410
3.193
0,061
0.127
0.916
0,980
Ours
1.311
3.314
0.062
0.123
0.952
0.981
'Ours (GF)' indicates our proposed model with gated-fusion.
Table 12: Performance Comparison on sub Test Set under Rainy Scenario.
Metries
Eval Distance
Method
MAE
RMSE
AbeRtel
RMSElog
61
0g
RadazNet
Singh al.
2023
1/159
3.535
0.099
0.175
0.918
0.967
GET-UP
Sun al.
2255
1.018
2.605
0.068
0.139
0.954
0.979
50m
Ours (GF)
0.926
2-146
0.063
0,130
0.958
0.982
Ours
0.891
2,421
0.061
0.128
0.960
0.982
RadarNet Singh et al.
22023
1.672
4.116
0.100
0,169
0.903
0,966
GET UP Sun el al.
M225c
1.204
3.162
0.070
0.145
0.950
0.977
70m
Oues (GF)
2.966
0.064
0.136
0.955
0.981
Oura
2.926
0.135
0.958
0.981
RadlaNeet Singh et
22023
1.750
4.361
0.100
0.185
0.913
0.964
GET-UP UP
Sun
M225c
1.275
3.392
0.071
0.148
0.949
0.977
80m
Oues (GP)
1.142
3.173
0.064
0.139
0.954
0.980
Ours
1.102
3.138
0.063
0.138
0.957
0.981
The results demonstrate that, under rainy conditions, our TRIDE model oulperforms the stale-of-the-arl GET-UP Sun et al. (2025ch by 12.48%, 13.46%, and 13.53% in MAE at evaluation distances of 50, 70, and 80 meters, respectively. Additionally, it is evident that depth estimation algorithms perform the worst under night-time scenarios. However, compared to GET-UP, TRIDE achieves RMSE improvements of 8.51%, 9.94%, and 10.21% at 50, 70, and 80 meters, respectively. Furthermore, replacing gated fusion with our proposed WaFB significantly improves night-time performance, further demonstrating the effectiveness of the proposed fussio method.
Interestingly, when comparing GET-UP and RadarNet Singh et al. [2023 while GET-UP shows significant overall improvements over RadarNet on the full test set, it does not outperform RadarNet in RMSE under night-time scenarios and even performs worse at the 80-meter range. These findings highlight that night-time remains challenging case for depth estimation. Nevertheless, TRIDE addresses this issue to some extent, demonstrating its robustness in such adverse conditions.
A.5
Convergence Curve of Loss Functions
In this section, plot the convergence curves of the losses used to train TRIDE. As described in Sec. 3.5 we employ two loss terms: classification loss and a depth estimation loss, the latter being composed of single-scan depth supervision and dense depth supervision. Figure 12 shows the evolution of these losses
during training. It is evident that the classification loss converges within a few thousand steps, whereas the depth losses require substantially more itcrations to stabilize. Morcover, the depth loss values are markedly
larger in magnitude than the classification loss; accordingly, in the final loss function, we we and wD during training.
23

---

Published in Transactions on Machine Learning Research (08/2025)
Table 13: Performance Comparison on sub Test Set under Night-time Scenario.
Metries
Eval Distance
Method
MAE
RMSE
AbsRtl
RMSEEog
d1
dg
RadarNet Singh et a
2023
2.343
4.686
0.150
0.235
0.840
0.935
GET-UP
Sun et a..
2025c
2.075
4.537
0.117
0.212
0.886
0.949
50m
Ours [GP)
2.009
4.443
0.118
0.199
0.883
0.953
Ours
1.876
4.151
0.109
0.189
0.898
0.955
RadarNet Singh et a..
2023
2.865
5.937
0.154
0.251
0.830
0,928
GET-UP
2025c
2.632
5.917
0.123
0.231
0.872
0,911
70m
Onns (GP)
2.582
5.631
0.124
0.213
0.889
0.947
Ours
2.364
5.329
0.115
0.203
0.885
0.949
RadarNet Singh el al. GET-UP Sum et al.
20223
3.014
6.340
0.156
0,255
0.827
0.926
2025c
2.791
6.356
0.125
0,236
0.888
0.939
80m
Ours (GE)
2.725
6.001
0.125
0.217
0.506
0,945
Ours
2.504
5.707
0.116
0.207
Loss Curve with Smoothing
Total loss
Classification loss
Single Depth
Dence Depth Loss
12
Los
100000
200000
300000
400000
500000
Step
Figure 12: Loss Convergence Curve.
A.6 Qualitative Results under Different Weather Conditions
In this section, we further analyze the qualitative results under different weather conditions. First, we show results under rainy scenarios. Then illustrate more results during night-time.
A.6.1 Rainy Scenarios
As shown in Figure 13 the first column displays the image with the projected single-scan ground truth depth, followed by the predicted depth maps from our TRIDE, RadarNet Singh et al. (2023), and GET-UP (Snn et al. (025cc at range of 80 meters. From the figures, is evident that our model predicts more detailed information. For instance, in the first, second, and fourth rows, TRIDE provides more accurate contours of humans. In the third row, model is the only others fail to distinguish between the upper part of the track and the sky. The last row demonstrates that our model produces more continuous depth maps compared to the others.
capable of correctly prodicting the track, while the
A.6.2 Night-time Scenarios
In this section, we analyze the predicted depth maps under night-time scenarios, illustrated in Figure As discussed in Sec 4.5 predicting accurate depth maps night poses significant challenges. Here, we compare the depth maps predicted by our TRIDE model with those from RadarNet and GET-UP.
Overall, our method demonstrates superior performance in detecting object shapes and identifying objects at farther ranges. For instance, in the first and second rows, TRIDE accurately captures the contours of the bus and cars. In the third row, our model the only one to correctly detect the bus at far distance. Additionally, TRIDE excels in capturing detailed information, as shown in the fourth row, where it accurately separates the human and the bus. In rostr the hther methods predict almost the same depth for both.
24

---

Published in Transactions on Machine Learning Research (08/2025)
Image with Ground Iruth
Ours
RadarNet
GET-UP
Figure 13: Qualitative comparison under rainy scenarios.
Image with Ground Truth
Ours
RadarNat
GET-UP
Figure 14: Qualitative comparison under night-time scenarios.
25

---

Published in Transactions on Machine Learning Research (08/2025)
despite the bus being much farther away in the image. Furthermore, our model produces more continuous depth predictions, as demonstrated in the fifth row.
A.6.3 Normal Weather Scenarios
Image with Ground Truth
Ours
RadarNet
GET-UP
Figure 15: Qualitative comparison under normal weather conditions.
26

---

Published in Transactions on Machine Learning Research (08/2025)
Here, we present additional depth maps under normal weather conditions, shown in Figure 15 From the figures, it is evident that our method predicts more continuous depth maps and effectively separates nearby objects. For example, in the second row, the upper parts of the two parked trucks are better distinguished compared to the other two methods. Additionally, the depth of the nearby parked cars in the last row is more accurately estimated.
For smaller objects, such as the human in the fifth row and the motoreycle in the seventh row, our method excels in capturing the shape of the objects with greater accuracy. Furthermore, at farther distances, method contimues to predict reasonable shapes and accurate depth values. For instance, in the ninth our method succossfully recognizes the bus at the farthest range, whereas the other methods fail to do so.
A.7 Failure Case Analysis
Image
Figure 16: Failure Cases Visualization.
In this section, we examine failure cases of our depth estimation, illustrated in Figure 16 We also compute the absolute error map between the predicted depth and the interpolated dense depth ground truth. As noted in Sec. 4.5 prediction accuracy degrades at night. In the first three rows of Figure 16 low image quality and challenging lighting prevent accurate shape prediction e.g., the motorcycle in the first row and the bus in the third row. We also observe that scenes lacking salient objects hinder reliable depth estimation, since the network has no reference points. Conversely, cluttered scenes with many vertical elements (street lamps, road signs, etc.) further complicate predictions. Addressing these issues remains important future work.
27

---

Published in Transactions on Machine Learning Research (08/2025)
A.8 Generalizability of the Text Branch
We introduce a text branch and integrate it, logether with the General Attention and Regional Attention modules, into an existing camera camera-radar radar depth estimation framework.
Specifically, we augment CaFNet Sun et al. (024b), an open-source end-to-end trainable model, while retain-
ing its original training strategy. In cont.rstt, most other methods [Singh et al. (2023 h Li et al. (2024a) Long
eet al. (2021) follow a two-stage pipeline, which complicates training, and as noted in Sun et al. (002cc), incurs
substantially longer training times. We therefore defer integration with those approaches future work. As
shown in Table 14 incorporating the text branch into CaFNNet yields clear performance improvement.
Table 14: Integrating Text Branch into Existing Model.
Model
MAE
RMSE
AbeRel
CaFNee Sun al. 20246
2.109
4.765
0.101
0.895
CaFNet
2.096
4.701
0.101
0.897
A.9 Runtime Analysis
Following WorDepth Zeng al. (2024a), generate all text descriptions offiine and persist them to disk. Subsequently, these text descriptions are encoded using frozen CLIP model, and the resulting text fea- tures are stored locally. This pipeline ensures that both text generation and encoding are performed only once per dataset, thereby eliminating redundant computation during model training and inference. In our architecture, the depth-essimatiin network's text branch begins with paragraph-level encoding (see Figure D) consuming the precomputed CLIP features as input. All timings were measured on a single NVIDIA A30 GPU. Text. generation requires 9.4 per frame, and CLIP encoding requires 0.235 s per frame. There- after, the standalone deptt-estimation network, including the text branch starting from the paragraph-level encoding, runs in 0.031 per frame.
A.10 Conclusion and Future Work
In this work, present TRIDE, novel approach that leverages text descriptions to support radar-camera depth estimation, achieving stale-of-the-arl performance on the nuScenes dataset. As highlighted in the previous sections, night-time poses significant challenges for accurate depth prediction. This underscores the importance of our proposed weather-aware fusion block, as images captured at night are less reliable, while radar remains robust under such conditions.
Future Work From the results in Table II Table 12 and Table 13 it is evident that depth estimation performs significantly worse at night-time, with MAE more than doubling compared to rainy scenarios. Improving depth estimation under night-time conditions therefore a critical area. for future research.
Additionally, in this paper, we propose tailored prompt design and leverage multimodal large language model for text generation, improving alignment between textual and visual features. Nonetheless, producing consistently informative descriptions for diverse perception tasks remains challenging. Future work may investigate the impact of such descriptions on other tasks (c.g., object detection, semantic segmentation) and develop methods to generate more lask-relevant lext. Moreover, reducing the latency reliable text generation will be essential support real-time applications.
28


Reviews:
[{"review_id": "CmatvJir5e", "reviewer": "Reviewer_FTjQ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This is a principled way to incorporate multimodal fusion with text. The fact that text supervision would contribute in a material way - especially, as it shows up as additional supervision makes sense. I am totally on board with the idea that a more granular set of cues adds richness to the scene rather than simply using bounding boxes or class info. \n\nSome of the design choices seem a bit arbitrary (perhaps that's too strong a statement), such as why would we want to combine text and radar? \n\nHow can we expect the text captioning network to predict depth? Why would we trust this? \n\nIn the setup, we predict text and then encode it as features (using an LSTM). But why not just take these features from the captioning network instead of transcribing, and encoding them all over again?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work proposes a multimodal camera-radar depth estimation setup, with inclusion of an extra 'text' modality to add additional supervision. By leveraging this extra modality, with cues to describe the scene in text, they claim that it improves depth estimation performance. \n\nThe architecture is somewhat detailed, but sensible. They generally follow existing ideas for radar camera fusion. For text, they present the image (scene) to a processing network which outputs a text description of the scene. This description is generated according to a prompt, which asks to describe the scene, together with an estimation of the depth. This text is then encoded into text features, processed with an LSTM and fused with radar. \n\nIn various places, cross attention is used for fusing modalities. There is a weather prediction head whose output is used in the decoder, taking into account information from all the modalities (radar is noteworthy in adverse weather).\n\nResults \nEvaluations are carried out on nuscenes and kitti, showing that their approach works. The ablations show the importance of adding multiple modalities.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See above. I would like justification for why we enrich text with radar. For instance, why not enrich camera also with it. \nIn nuscenes, we don't have many adverse weather scenes (as far as I know). How do the adverse weather fittings of this network help here? \nIn some other work (e.g. RC-PDA), it is seen that radar height measurements from nuscenes sensors are not reliable. It necessitated an elaborate set of operations to clean up this mismatch. Do the authors use any such preprocessing or cleanups in this work? The point I would like to understand is whether the need for such cleanups is obviated in the present work because of evolutions in architecture (I would think not).", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "SzWuSZsVgi", "reviewer": "Reviewer_yVLu", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- This is the first study to utilize text information in radar fusion, and there are few studies focusing on weather adaptation, making this research inherently beneficial for practical applications.\n- The authors proposed unique blocks such as REB and WaFB and demonstrated their effectiveness.\n- The evaluation is comprehensive.\n- The evaluations are presented in Tables 10-12 for normal, rainy, and nighttime scenarios, demonstrating consistent effectiveness in each scenario. Furthermore, the fact that the nighttime scenario is more difficult than the other scenarios provides a useful perspective for future research. This would simply strengthen the work in my view.\n\nWeaknesses:\n- Please refer to 2 and 3 in Requested Changes below. Although these are minor, they could undermine the reliability of the paper. The authors should check the content thoroughly.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes TRIDE, a new depth estimation network that fuses three modalities: images, radar, and text. Inspired by recent radar and image fusion research, this study focuses on the fusion of three modalities, whereas previous studies have been limited to two modalities, and multi-modal learning that considers text has not been proposed.The contributions of this study lie in its focus on the fusion of these three modalities, particularly the use of text encoders (sentence-level features), the proposal of the Radar-Enrichment Block (REB) and the Weather-aware Fusion Block (WaFB), and the effective integration of these components to maintain consistency. The effectiveness of the proposed method is evaluated using standard radar point cloud datasets such as nuScenes.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. I recommends including Table 10-12 in the main text.\n2. Although Fig. 1 is useful for understanding the overall purpose, it is not referenced anywhere. It would be helpful to clarify which blocks in Fig. 1 correspond to the proposed REB and WaFB blocks mentioned in the introduction. Please cite it correctly in the introduction. In addition, there is inconsistency in terminology between Fig. 1 and Fig. 2. For example, the terms 'Radar Points Projection' and 'Text Encoder' in Fig. 1 are not included in Fig. 2. 'Depth Estimation Decoder' is simply referred to as 'Decoder' in Fig. 2. \n3. Table ?? is included on page 9.\n4. I strongly recommends that the source code be made publicly available. This will enhance the credibility of the paper", "broader_impact_concerns": "It may be worth mentioning that MLLM occasionally misdetects, which can cause significant deviation in depth estimation and increase the risk of collision. I recommends explicitly stating that the introduction of a fail-safe mechanism and uncertainty estimation is essential.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "LTOPbdfVYk", "reviewer": "Reviewer_TpFQ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengthens:\n1. The proposed method achieves promising performances on the two leveraged dataset, demonstrating the superior performance to achieve robust depth estimation by using three modalities. The proposed method could benefit the community.\n\n2. The authors have conducted ablation study to illustrate the effectiveness of each propose module.\n\n3. The paper is generally well-written and easy to understand.\n\nWeaknesses:\n\n1. The authors propose to use LSTM to achieve paragraph level feature fusion, how about direct fusion using addition, concatenation, and multiplication or using transformer block to encoder it? The authors are suggested to provide more ablations regarding this design,\n\n2. Could the authors provide some quantitative measurement regarding the text-radar alignment of their trained model regarding the Radar-Enrich Block? or could it be visualized somehow?\n\n3. Lack of ablation regarding the radar feature extraction backbone, can it work well for other architectures, e.g., VoxelNet?\n[1] Zhou, Y., & Tuzel, O. (2018). Voxelnet: End-to-end learning for point cloud based 3d object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 4490-4499).\n\n4. The authors are suggested to add more formulas in Sec 3.4 to make the description clearer\n\n5. The authors are recommended to show the convergence trajectories based on these two loss functions.\n\n6. The authors are suggested to double check if all the hyperparameters are mentioned in the implementation details section. The weights of the two losses are not mentioned in the implementation details.\n\n7. Most of the existing approaches do not rely on the textual description generation, I am wondering how about the performances if we simply add text branch on them, e.g., using text branch on GET-UP, could the proposed method outperforms this variant?\n\n8. Regarding the runtime of the proposed method, did the authors take the description generation part into consideration? The authors are also encouraged to add more analysis regarding the runtime.\n\n9. Regarding the quantitative results analysis, the authors are encouraged to conduct anaylsis regarding why their method can achieve such good performances based on the novel designs introduced. More in-depth analysis should be provided.\n\n10. Regarding the evaluation on KITTI section, the table is not correctly citated, the authors are also encouraged to add more analysis.\n\n11. Failure cases are suggested to be provided in the qualitative results to enable limitation analysis.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes TRIDE, a novel multi-modal framework that integrates radar, image, and text information to enhance depth estimation, particularly under adverse weather conditions. A tailored text generation strategy produces both global and region-specific descriptions, which are encoded and fused into the model using attention mechanisms. To improve robustness, the authors introduce a radar-enrichment block that refines text features with radar data, and a weather-aware fusion block that adaptively balances sensor inputs based on environmental conditions. TRIDE achieves state-of-the-art results on the nuScenes dataset, demonstrating notable improvements in both accuracy and efficiency over existing methods.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1.The use of LSTM for paragraph-level feature fusion is interesting, though it may benefit from additional justification. It would be helpful to include comparisons with simpler alternatives such as addition, concatenation, or multiplication, as well as with transformer-based encoding. These comparisons would strengthen confidence in the current design choice.\n\n2.The Radar-Enrich Block is a key component of your model. Including some form of quantitative measurement or visual evidence of the text-radar alignment would help validate its contribution and provide a clearer understanding of the fusion effectiveness.\n\nIt would be valuable to investigate whether the proposed approach maintains its effectiveness when used with different radar backbones. For instance, testing with architectures like VoxelNet [1] could demonstrate the generality of your design.\n\n3.The current description in Section 3.4 may be difficult to follow for some readers. Adding more detailed mathematical formulations could help improve clarity and ensure a more precise understanding of the proposed mechanism.\n\n4.Including convergence curves for the two loss functions used in training would provide helpful insight into the model’s optimization behavior and training stability.\n\n5.Please double-check that all hyperparameters are clearly stated in the implementation details. For example, the weights used to combine the two losses do not appear to be mentioned and should be added for completeness and reproducibility.\n\n6.Since most existing approaches do not rely on textual descriptions, it would be interesting to see how performance changes when simply adding a text branch to existing methods like GET-UP. Such a comparison could help clarify the unique contribution of your fusion strategy beyond just the inclusion of textual data.\n\n7.It would be appreciated if you could clarify whether the runtime figures reported include the time for generating textual descriptions. Providing a more detailed runtime analysis would also help readers better understand the trade-off between performance and computational cost.\n\n8.The results would benefit from deeper analysis explaining why your design leads to strong performance. Connecting the observed improvements more explicitly to the novel aspects of your model would enhance the reader’s understanding of its effectiveness.\n\n9.In the section evaluating KITTI, the table citation appears to be incorrect. Additionally, providing a more thorough discussion of the results in this section would make the evaluation more informative.\n\n10. Including a few failure cases in the qualitative results would allow for a more balanced view of the model’s limitations and could help guide future improvements.", "broader_impact_concerns": "There is no broader impact concerns for this paper.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)