Anytime Depth Estimation with Limited Sensing and Computation Capabilities on Mobile Devices
Authors: Yuedong Yang
Zihui Xue
Radu Marculescu
Venue: CoRL2021 Poster (2021)
Keywords: Depth Estimation
Anytime Algorithm
Energy-aware Optimization
Mobile Devices

Abstract: Depth estimation is a safety critical and energy sensitive method for environment sensing. However, in real applications, the depth estimation may be halted at any time, due to the random interruptions or low energy capacity of battery when using powerful sensors like 3D LiDAR. To address this problem, we propose a depth estimation method that is robust to random halts and relies on energy-saving 2D LiDAR and a monocular camera. To this end, we formulate the depth estimation as an anytime problem and propose a new metric to evaluate its robustness under random interruptions. Our final model has only 2M parameters with a marginal accuracy loss compared to state-of-the-art baselines. Indeed, our experiments on NYU Depth v2 dataset show that our model is capable of processing 224$\times$224 resolution images and 2D point clouds with any computation budget larger than 6.37ms (157 FPS) and 0.2J on an NVIDIA Jetson TX2 system. Evaluations on KITTI dataset under supervised and self-supervised training show similar results.

Anytime Depth Estimation with Limited Sensing and Computation Capabilities on Mobile Devices
Yuedong Yang
Zihui Xue
Radu Marculescu
The University of Texas Austin
{albertyoung, sherryxue, radum @ @utcxas.cdu
Abstract: Depth estimation is a safety critical and energy sensitive method for environment sensing. However, in real applications. the depth estimation may be halted at any time, due to the random interruptions or low energy capacity of bat- tery when using powerful sensors like 3D LiDAR. To address this problem, we propose a depth estimation method that is robust to random halts and relies on energy-saving 2D LiDAR and a monocular camera. To this end, we formulate the depth estimation as anytime problem and propose a new metric to evaluate its robustness under random interruptions. Our final model has only 2M parameters with marginal accuracy loss compared to oate-offttheart baselines. Indeed, our experiments on NYU Depth v2 dataset show that our model capable of pro- cessing 224 resolution images and 2D point clouds with any computation budget larger than 6.37ms (157 FPS) and 0.2J on an NVIDIA Jetson TX2 system. Evaluations on KITTI dataset under supervised and self-supervised training show similar results.
Keywords: Depth Estimation, Anytime Algorithm, Energy-aware Optimization, Mobile Devices
Introduction
Depth estimation plays a critical role various applications, such as autonomous driving, aug- mented reality and virtual reality. While attempting measure the distance from camera to obsta- cles, existing approaches resort to depth sensors like LiDAR or structured-light sensors. The goal is to reconstruct the depth in full resolution utilizing images, along with sparse depth measurements.
variety of model architectures [1-12] have been proposed to effectively fuse the RGB images and LiDAR point clouds (PC). However, most approaches cannot be directly deployed on small robot platforms or mobile devices, say a mobile phone or AR glasses. This due to multiple factors: First, these methods are passive to the changes in the environment. In contrast, a dynamic behavior is must for a real robotic system. For example, when an obstacle appears suddenly, it is preferably to sacrifice accuracy for fast response avoid crashing. Second, many methods rely on bulky, heavy and high-end LiDAR systems that consume much power to generate dense and accurate PC [13-15]. On the contrary, depth sensors installed edge devices may produce extremely sparse and non- uniformly distributed depth patterns, e.g., a depth line produced by cheap 2D LiDAR as shown in Figure 1. The sparsity pattern generated by 2D LiDAR very regular such that the PC can be represented with an 1D vector, while other types of PC in Figure ((b) can only be represented with high-dimensional arrays. Moreover, boost performance, deeper and more complicated models are adopted together with computation-intensiiee algorithms [5, 16]. This usually requires powerful GPU clusters and those are not available in robotic systems based on edge devices.
In this work, we argue for a more realistic problem setting for edge computing, namely anytime depth estimation with monocular camera and 2D LiDAR on small robots and mobile devices con- sisting of resource-ling limited embedded devices. Consequently, in addition to the depth prediction ac- curacy, we need optimize for performance under highly dynamic environment, limited sensing and computational capabilities. In fact, all these constraints are very relevant in many common robotic systems, e.g., the Pioneer and K5 Security Robot; they operate in dynamic environments, yet come only equipped with 2D LiDAR and battery- embedded computers [2].
As our main contribution, we propose new metric called anytime loss, and optimize anytime depth estimation model robust to random halts that requires only an energy-efficient 2D LiDAR and
5th Conference on Robot Learning (CoRI. 2021), London, UK.

---

RGB Image
2D Point Cloud (PC)
Time
Ums
Stop!
(0.4235
0303
Dense
Umitoral Sparse
Mono
2D LIDAR
Lakest
Camera
104181
Degth
2/2
Struharalyy Sparse
Depth Map
Pinal 12.2ms [0.415]
(a) W Workflow
(b) Point Clouds and Depth Map
Figure 1: (a) Example workflow and timeline of depth estimation model. As computation progresses, increasingly accurate depth maps are generated (numbers in brackets denote RMSE). Inference can be halted any time (e.g,, when deadline reached or more urgent task needs to start) and the best estimation returned. (b) Point clouds (PC) with different sparsity structures and the output depth map.
monocular camera. We design the network architecture as two-stream encoder-decoder with an error correction unit (ECU) placed in middle (Figure 2). We optimize the encoder by utilizing the sparsity in 2D PC shown in Figure yielding light-weight encoder with marginal accuracy loss. We optimize the decoder by introducing early-exit paths. Both methods are proved to be effective theoretically and practically improving model's dynamic performance under random halts.
Exhaustive experiments on NYU Depth v2 and KITTI, under both supervised and self-supervised training. show that our method outperforms most state-of-the-art baselines by taking the highly non- uniform depth measurements into consideration with significantly smaller model sizes. Furthermore, without resorting to any quantization and pruning, we show that our model can perform real-time depth estimation on an NVIDIA Jetson TX2 system operating with a computation budget larger than 6.37ms (157FPS) and an energy figure of only 0.2J.
2 Related Work
2.1 Depth Estimation with Camera and LiDAR
Eigen er al. [17] introduce convolution neural networks (CNNs) to the problem of depth estimation. Sparse convolution was then proposed to address the variant sparsity of point clouds [14]. Jaritz et al. [18] design an encoder-decoder network to accomplish both depth estimation and semantic segmentation with only the last layer changed. However, these methods simply rely on network architectures to address various structures of point clouds. The performance of a straightforward network drops significantly when the sparsity of point clouds increases [1, 18]. Methods tailored for different sparsity of point clouds are thus proposed. Based sparsity level of point clouds, camera- LiDAR methods can be divided into three categories: dense, uniformly sparse, and structurally sparse, illustrated in Figure (bb) Our work focuses on the most challenging third setting, i.e., depth measurements are sparse and highly non-uniform.
Dense Point Clouds. For the dense point clouds in Figure 1(b), the surface normal can serve as a good intermediate feature. More precisely, the density point clouds ensures that every point has enough data from its neighbourhood; this leads to an accurate estimation of the surface normal. Zhang el al. [15] propose to first estimate the surface normal and then combine with PC for RGB- D depth completion. Qiu el al. [13] extend this idea to the camera-LiiAR sensor setting. After the surface normal estimated, their method generates two depth estimation results with (image, PC) and (surface normal, PC) taken inputs, respectively. The final result is obtained by fusing the two depth maps with an attention mechanism.
Uniformly Sparse Point Clouds. The uniformly sparse setting shown in Figure 1(b) refers to the scenario where very few points are uniformly scattered across projected plane. Jaritz et al. [18] claim that naively applying CNNs to sparse data can easily fail since CNNs are sensitive to missing data. address the problem, they propose to use a NASNet encoder [19] to encode the image and
sparse depth plane separately. The encoder for the depth plane acts as depth completion unit to densify the sparse depth map without referring to information from the image plane. Ma et al. [1] propose single regression network to learn directly from raw RGB-D data, and explore the impact

---

of the number of depth samples on prediction accuracy. Chen et al. [5] propose a convolutional spatial propagation network (CSPN) learn the affinity matrix for depth prediction.
Structurally Sparse Point Clouds. The structurally sparse PC illustrated in Figure 1(b) is the most challenging scenario where few LiDAR points are structurally seattered across line few fixed locations. While this is commonly produced by depth sensors in low-cost devices, such as 2D LiDAR on Hloor-seeping robots, little research targets this setting. For instance, [2] and [11] propose to estimate a dense depth map with a 2D LiDAR by stacking the LiDAR points along the gravity direction and then estimate the depth deviation from the generated surface to the actual object surface. However, they rely on the assumption that the depth of an actual object surface is similar to the generated surface, which does not often hold true in open spaces. Our proposed method provides better ways of utilizing sparse 2D points rather than simply stacking them.
2.2 Depth Estimation with a Monocular Camera
In recent years, deep learning methods have been considered for depth estimation using monocular camera due to its low implementation cost. As the general trend has been to rely on deeper and more complicated networks in order to achieve higher accuracy, the ofate-of-theane depth estimation algorithms do so at the cost of increased computational complexity. To achieve real-time inference on embedded system, Wofk al. [20] demonstrate FastDepth, a low-latency, high-throughput monocular depth estimation method that can run on embedded systems. Nekrasov el al. [12] propose method performing semantic segmentation and depth estimation jointly with single model.
2.3 Self-Supervised Depth Estimation
Recent work on sulf--upe depth estimation [21-23] shows great potential to effectively make use of unlabeled data. Indeed, the self- sef-ssuperviee methods can achieve comparable or even better performance than supervised ones. Our work focuses on the model itself and addresses to following fundamental question: Can ones make the depth estimation model satisfy strict power/performance constraints, yet be practical in real-worlh applications? In the experimental section, we show that our model can indeed work very well with state-of-the-ane self-supervised training framework.
3 Problem Formulation
3.1 Depth Estimation with Monocular Image and 2D Point Clouds (PC)
With an input RGB image and 2D PC, the depth estimation model needs to predict the depth for every image pixel, while yielding depth map with the same shape as the input image. Depth prediction must finish before the model runs out of the computational resources, egg. time and/or energy. The PC generated by 2D LiDAR consists of a set of points in plane. We project the PC points to the image plane, and then sequentially concatenate the depth value of these projected points into an ID vector C. An image is a 3D array with two spatial dimensions. The output depth map D is 2D matrix with the same spatial shape of the image; each element in D is the predicted depth for the corresponding pixel in the image, the computation budget B is scalar indicating e.g. how much time or energy is available to estimate the depth. In the end, we formulate our depth estimation model as function D B)
3.2 Anytime Depth Estimation
In robotic systems, due to the dynamic nature of the real world application, the allowable compu- tational budget (i.e., time and/or energy) B varies with time. We cannot know the computational budget in advance and the model has no control over it; to simplify the problem formulation, we assume that the available computational budget B E (0, Byaaz) is random variable and, before every inference, the exact budget is sampled from it. Intuitively, the objective of an anytime algo- rithm is to get satisfactory output no matter how much computational budget is available. With the loss function I(-), the overall objective of anytime algorithm is to minimize the anytime loss, i.c., the expected loss under the budget distribution: L(ʃ)= Er,c, ( ( (I, CI, C)).
To make the anytime loss practical, we define the discrete budget anytime algorithm, iee., if there exists a partition P for (0, ) such that VP, E P and BB,, Bg E P., [(,, B,) = =[(T, C, Bş),
3

---

Budget Distribttion
Sumpler
Evaluate Li
Design Stage
12-2as
(0A15)
Halt Control
[P] Glms
Resource Monitor
Deploy Stage
Bodget
Model
(0.418)
2D CNN
(F. F.
1.44ss
Path
Fing
Img Enc
Tmg Enc
ECU
Dec
Dec
Dec
(0
F
D
F
(0:033
CID CNN
PC
1x1 Cozr
Path 3
Enc
Coav
Path
6,37uu
PC
1x1 ConY
Path
(0
Nearest Interpolate
Concatecate
Backup Path
Path
Figure 2: Proposed network architecture. First, two dedicated encoders map the input image and point clouds C (PC) to hidden features F, and Fc respectively. Then, emor correction unit (ECU) fuses feature these two modalities. Finally, shared decoder upsamples the ECU output. the design stage, we sample from simulated computational budget distributions, e.g., time and energy distributions, halt the inference when the model hits the budget and evaluate the performance with our proposed metric L(f). f) At the deployment stage, monitor the system and halt the inference when necessary. Example shown right annotated with elements for loss T.(f) calculation: exccution time, accuracy loss RMSE in brackets and length of each interval each budget partition.
representations
we say the model interval that corresponds to the partition i. Since for any B P. the output of f(I, C, B) is con- f(I, C, B). The model can be rewritten into: C), = EPPl Ip, where Ip, (B) is the indicator function, i.e. Ip. (B) = iff B E Ps otherwise Ip. (B) O. We assume that the computational budget random variable B obeys a uniform distribution U0, Bmaw), ) then the optimization objective can be rewritten as:
discrete budget anytime model. Here the [Pd] is the length the budget
stant, we can define new function fI,
[P]
Bmar
L(f) = F,,.C, CBB) =
(.) () (bo)) = b]] db
(1)
IPI Σ C))PP B Bmax 1=1
where |P]] is computed for each budget interval across partition P. Note that, all these budgets are defined under the assumption that the system configuration kept the same, i.e. the CPU clock frequency is set to run at the maximum allowable frequency.
Architecture Design and Optimization
4.1
Motivation
The intuition behind our design closely aligns with problem setting. Consider the diserete budget anytime optimization objective L(S) proposed in the previous section. Here we propose two ways to optimize the objective:
1. For each term (T, Pa), if we can minimize the budget required at each stage from [Pa to IP:l < |P] by changing the model from fü to for while keeping ([, C)) x I(I(TT then it is expected that [(f,(T, P3) s T(Fa(T C)) and then L(J) is optimized.
2. If we can split each term ((f(I, C))] into sub-terms I(fi, (I, C)))Ps with E, IP,, | [P.
but ((f;, C)) < ((fa(T, C)), then E, (T, C)) I(fr(T, C))|B, and T.J) is optimized.
4.2 Model Optimization Based on U-Net
We adopt the U-Net [24] style network architecture, shown in Figure 2, where the model D = f[I,C. can be partitioned into an encoding part: F fIne(I, B) and a decoding part D faee F, B). in which F is the hidden feature. The encoder and decoder are optimized as discussed above. For the encoder Sencs we design cross-modality encoder with <1.5M parameters by exploiting the low dimensionality in 2D PC and alignment-based data fusion module. For the de- coder fdees we generate coarse, but low-latency depth maps on-the-fly by introducing budget-aware light early-exit paths into the model. The details are discussed next.

---

To design a cross cross-modality encoder, there are two problems to solve: How to extract relevant features and how to fuse them? Most previous methods [1. 5, 13, 25, 26] design feature encoders to map the image and PC to shared feature space with two spatial dimensions; this because they work with dense or uniformly sparse PC, where both modalities provide information of the global region, as shown in Figure ((b). In such case, the feature encoders for both modalities are similar and data fusion is as simple as adding or concatenating features from the two modalities. However, as Figure 1(b) shows, the case of 2D PC is different. Depth measurements in 2D PC only correspond to limited area of the entire image plane. In problem formulation section, we represent the PC with an ID vector. So instead of trying to extract feature with two spatial dimensions based on an ID vector, we keep the PC feature fow-dimensional with only one spatial dimension. Moreover, because the image and PC features have different spatial dimensions now, conventional fusing methodologies are not applicable here; so we also propose an alignment-based feature fusion module to fuse the features from image and PC with different spatial dimensions.
Encoder Optimization:
As Figure shows, features Fy and Fc are extracted with two dedicated encoders and fused with an error-correction unit (ECU). Both encoders keep the same spatial dimension before and after encod- ing, 2D for the image and ID for the PC. The ID feature extraction requires much less computation than the 2D feature extraction, so the computation budget is saved and [P]] is reduced.
ECU fuses F1 and Fc by aligning features at spatially corresponding locations. Points in PC are aligned with the neighboring pixels in the image. We assume this is also true for features extracted from the image and PC. As shown in Figure 3, the ECU extracts the neighboring image features and compares it with the PC features. Based on this comparison, the ECU generates the error compensation value and applies it to the image feature.
With a vanilla decoder, before the decoder finishes computation, the best depth map can get is generated by the PC nearest interpolation. The high loss value together with the large computational budget required by the entire network contributes greatly to the loss L(f).
Decoder Optimization:
To reduce the loss when the inference is halted half-way, we export the intermediate features from different stages of the decoder and convert the feature maps to the depth maps with small CNNs. For example, in Figure 2, path 2 extracts the intermediate feature generated by the first-stage decoder, and converts the feature map to a depth map with an 1 xx 1 convolution layer. Compared with the interpolated depth map, path 2 returns more accurate depth map; compared with the depth map generated with entire decoder, path 2 requires a smaller computational budget. As the computation progresses, depth maps generated with intermediate features become increasingly more accurate. Here, by using early-exit paths, we introduce a new axis in the quality-speed trade-off space. Con- sidering that the feasible positions for inserting an early-exit path is limited, in practice, we optimize the decoder by manually inserting a few early-exit paths and evaluating on various datasets.
4.3 Network Architecture
4.3.1
Light-weight Encoder: Backbone
We design two encoders that map input images and PC into their hidden feature spaces respec- tively, as shown in Figure 2. While targeting low latency, we consider MobileNet v1 [27] as our encoder of choice. To further reduce inference time and memory requirements, we remove the last 9 computation-heavy convolution layers of one standard MobileNet while keeping the down-scaling factors unaffected, which reduces the parameter count and computation of our image encoder from 2.94M to 1. 1.08M and from 0.582 GMACs to 0.305 GMACs, respectively. Our PC encoder is a fully convolutional network with ID convolutions. We evaluate the effectiveness of our design in Section 5. The most obvious benefit of our PC encoder design is that the ID convolution costs very few parameters and computations, with only 44k parameters and 0.966 GMACs.
4.3.2 Light-weight Encoder: Error Correction Unit (ECU)
We train a CNN to perform the feature alignment mentioned in the encoder optimization section. Feature Fe from PC carries accurate information of a limited region. If we align the ambiguous image feature F, with Fc, then a more accurate global depth feature can be expected. As Figure shows, ECU contains two parts: alignment error extraction and alignment error compensation.
Alignment Error Extraction: The mismatch between the features Fc and F, is equivalent to the mismatch between Fc and the spatially corresponding region in F₁. The ECU extracts the
5

---

spatially corresponding region by cropping 3-pixel-wide band around the projected positions and then summarizes these regions into feature with one spatial dimension via convolution.
The summarized image region feature has the same spatial dimension with the PC feature now. We extract the alignment error feature by concatenating and passing them through an ID point-wise convolution layer.
Eng
Aligrment
Compereation
F,
Neighbor Region PC
Aligmment Error Compensation: Given the alignment error feature, we use an ID convolu- tion layer generate the compensation value while has only one spatial dimension. We broadcast the compensation the image fea- ture by nearest interpolation.
Image Featuse
Crop Conv
Concat
Pombense Conv
Cone
Interplaaton
F
PC Feature
4.3.3 Anytime Decoder with Early Exit Paths Figure The Error Correction Unit (ECU) design.
We design our anytime decoder based on FastDepth [20]. The decoder is composed of five up- sampling layers, where each up-sampling layer consists of a depth-wise convolution layer and an interpolation layer. After some selected decoder layers, we insert early-exit paths to generate the depth with intermediate decoder features. To make sure that the early-exit paths would not lead to large computation overhead, the early-exit paths in our decoder contain only one or two point-wise convolution layers. Moreover, compared with Fast and considering that the feature from our encoder has a better quality, we reduce the numbers of decoder channels; by doing so, the parameter count of our decoder drops from 0.809M to 0.525M.
5
Experimental Results
5.1 Experimental Setup
Datasets. We set up datasets and accuracy evaluation metrics NYU Depth v2 [28] and KITTI Odometry [29] datasets following Ma et al. [1], except that for the KITTI Odometry dataset, we only use left-view images. To simulate the PC generated by a 2D LiDAR, we sample line of points at the center of the ground truth depth map.
Implementation. The network is implemented in PyTorch and trained with 16-bit float point pre- cision. For training, the batch size is set to 96 and the learning rate is initially set to 0.05 then decreases to 0.02 with polynomial learning rate decay scheduler. The optimizer is SGD with an 1 10-4 weight decay. Finally, the encoder MobileNet
pretrained on ImageNet [30].
Deployment. We deploy our network on both Raspberry Pi 4 and NVIDIA Jetson TX2 with TVM [31] and evaluate the speed and energy consumption under random halts. The power of Jetson TX2 and Raspberry Pi 4 is measured on the power supplier with INA226 [32] under 50 Hz sample rate. No additional network pruning or quantization is used in the deployment.
Baselines. Every open-sourced baseline trained with its original training method. We only change the input PC from dense uniformly sparse PC 2D PC. For methods compatible with TVM, in order to mimic an anytime algorithm, we change the input image resolutions, fine-tune the model for each resolution and run them in parallel. For fair comparison, the parameter count and flops reported for each resolution only consider the corresponding model, instead of all of these models.
5.2 Results on NYU Depth V2 Dataset
Table presents the quantitative evaluation results in terms of accuracy, model size and inference speed. Figure visualizes the depth maps predicted by each path in our model. Results from different paths are generated in only one with the same model. For fair comparison in terms of image resolution, we also evaluate our method with 224 x 320 images. As shown in Table 1, no significant difference in accuracy is found.
Compared with most methods in Table I, our method achieves comparable or even better precision with an order of magnitude smaller model size. For instance, using only 6.4 ms and 0.3 GFlops, path1 returns a depth map with better quality than EneDecNet [8]. the computation progresses, depth maps with better accuracy are predicted. The latest depth map generated by path 4 (at

---

RGB Image
0.0ms Interpolation:
6.4ms
8.0ms
9.8ms
12ms
Ground Truth
Gemerated by the noural networis in our model
Figure 4: Anytime predictions on NYU Depth v2. The depth maps are generated step-by-ster in one run with the same model. The first prediction at 0.0ms is generated with nearest interpolation, if the neural network is halted before making any prediction. Predictions from 6.4ms to 12ms generate more details in the depth map as computation progresses. For more examples, please refer to the Section S. of our Supplementary Material.
Comparison with prior work in terms of accuracy, lateney and network size the NYU Depth v2 dataset. Latency measured on the NVIDIA TX2 system. For δ1, the higher, the better; for other variables, the lower, the better. "Param" is short for parameters count (in millions). "NA" means the method not open-sourced or not supported by TVM. Our experiment with a different input size is presented in the last row.
Table 1:
Method
Input size
Lateney
RMSE
Rel
51 t
Param
GFlops
FastDepth [20]
224x224
17.1ms
0.583
16.4%
0.767
4.016
0.867
Parse [2]
256x320
NA
0.442
10.4%
0.878
NA
NA
S2D [1]
228x304
554.7ms
0.426
10.0%
0.885
28.39
8,910
EncDecNet [8]
x00
NA
0,635
15.5%
0.775
0,484
1.276
CSPN [5]
208x
NA
0.379
7.91% %
0.916
218.122
261.746
PathO
224x224
0.0ms
0.9871
27.90%
0.6605
0,000
0.000
Path1
224x224
6.4ms
0.4651
11.52%
0.8598
1.489
0.319
Ours
Path2
224xX244
8.0ms
0.4225
10.01%
0.8871
1.983
0.423
Path3
224xX24
9.8ms
0.4184
9.80%
0.8906
2.006
0.496
Path4
224x224
12.2ms
0,4156
9.80%
0.8909
2.014
0.594
Ours
Path4
224x320
16.42ms
0.4191
9.84%
0.891
2.014
0.848
12.2ms) has lower RMSE than all baseline method except CSPN. But considering the two orders of magnitude difference in computation counts, our solution is more suitable for edge devices.
Loss (tfi0 c)): RRMSE [metel]
1.0
TX2
Loss MFA0 C)): RMSE [meter]
Ours-TX2
0.9
PathO
FiniDDoth- TX2
Roid
0.9
FastDepth TX2
Ours Rpi4
RastDepth-RRid
S2D-TX2
0.7
B max
0.7
B max
Path1
L(f)
Path2
0.5
Path3
0.5
Path4
0.4
6
8110
50 100
500
0.1
0.2
0.3
0.4
0.5
Budget B: Time [ms] (Log)
Budget B: Energy U]
Figure 5: NYU Depth v2 results with different computational budgets. Every drop the an optimized prediction made by model: the area under curve equals the anytime loss L(f) according to Equation 1. Path 0 generates a backup depth map with the highest RMSE (0.9871) by interpolation, if the neural network is halted before making any prediction, while results on other paths are predicted by neural networks. Predictions requiring more computational budgets than the maximum budget Bmar considered be ineffective predictions. For instance, for S2D with maximum 0.5J energy budget, only paths 0-2 effective since paths 3-4 require 0.954J and 6.244J respectively (exceeding the maximum energy budget 0.5J). For more detailed experiment results, please refer to the Section S.3 in our Supplementary Materials.
corresponds
Anytime Evaluation. We evaluate the dynamic performance of our model with random computa- tion budgets caused by random interruptions. Evaluations are based on our anytime loss proposed in Equation 1, expected accuracy loss under random available computational budgets. RMSE is used as the accuracy evaluation function I(-) in Equation 1. Two computation budgets, i.e., time and energy, are considered in our experiments. Experimental results are listed in Table 2 and illustrated in Figure 5. Since other baseline methods contain custom operators not compatible with TVM, we only evaluate method against FastDepth [20] and S2D [1].
7

---

Our method outperforms all the baseline methods on NVIDIA Jetson TX2 system, but fails on Raspberry Pi 4 when the maximum budget becomes too small to make fair comparison. Figure provides the intuition behind these results since every drop the curve corresponds to an optimized prediction and the area under the curve equals the anytime loss [(()), according to Equation 1.
Table Anytime loss evaluated with time and energy computational budgets. Lr @33ms stands for loss L(F) with maximum 33ms as budget. E stands for Energy, "NEE" for No Barly-Exit. Bold shows the best results.
Method
Device
1r0000ss
Q50ms
L Q333s
Lg00.5J
Lg R0.251
FastDepth [20]
TX2
0.601
0,624
0.648
0.738
0.898
S2D [1]
TX2
0.604
0.740
0.869
0.763
0,987
Ours-NEE
TX2
0.484
0.554
0.626
0.773
0.891
Ours
TX2
0.453
0.490
0.529
0.654
0.891
Fast [20]
Rpi4
0.803
0.906
0.979
0.726
0.809
Ours
Rpi4
0.755
0.987
0.987
0.582
0.746
Our first observation is that for most halts, our model returns depth map with a better quality, i.e. shown in Figure 5, the curve of model is always below others; this is achieved by exploiting the sparsity structure in 2D PC and designing cowdimensional encoders. Second, if we remove all the early exit paths our method, there is significant increase the anytime loss, especially when the maximum budget is small ("Ours-NEE" in Table 2). Since for any halt happening before our model makes any prediction, we can only resort the interpolation based backup path, path 0 which nearly doubles RMSE compared even with the worst result generated by neural networks. Taking experiments on TX2 as an example, by inserting early-exit paths, we reduce the budget required for predicting the first depth map from 12.21ms to 6.37ms, so for nearly half of the halts happening before we finish the entire inference, a better depth map is returned.
5.3 Results on KITTI Odometry Dataset
Table 3 compares the performance of our approach with prior work on the KITTI Odometry dataset. We evaluate our model under both supervised and self-supervised training, where self-supervised training is based on MonoDepth2 [21]. Our method achieves comparable accuracy in terms of relative error and 81 score, with orders of magnitude fewer parameters and computations. For more detailed results, see Section S.2 in the Supplementary Materials.
Table 3: Results on KITTI Odometry dataset. First five rows correspond to "Suuperviedd", while last two correspond to "Self-supervised.. To save space, we only show the result of Path 4. Bold shows the best results. "Param" is short for parameters count (in millions). "NA" means the method is not supported by TVM.
Method
Input size
Latency|
RMSEL
Rell
81 f
Param
GFlops
S2D [1]
228x912
374.7ms
4.272
10.0%
0.902
11.49
8.593
EncDecNet [8]
228x912
NA
5,462
11.6%
0.839
0.484
4.000
CSPN [5]
228x912
NA
3.661
6.81% %
0.735
218.1
261.7
PENet [16]
352x1216
NA
2.694
11.4%
0.942
131.9
405.3
Ours Path4
224x896
40.45ms
4.183
8.06%
0.891
2,014
2.375
MonoDepth2 [21]
192x640
77.26ms
3.259
8.16%
0.926
15.24
8.051
Ours Path4
192x640
25.85ms
3.608
8.13%
0.943
2.014
1.518
Conclusion
In this paper, we have presented a novel anytime depth estimation method on mobile devices, which robust to random halts and works with power-savine but cappagaaaaaaaaliiiiniiei limited 2D LiDAR and monocular camera. We formulated our problem as an anytime depth estimation problem and proposed a new metric comprehensively evaluate robustness under dynamic interruptions. Based on our proposed metric, we have designed and optimized our model by exploiting the sparsity structure in 2D PC and introducing early-exit paths. Experiments on the NYU Depth v2 and KITTI datasets on edge devices show that, with only 2M parameters, model has comparable accuracy and better robustness to random halts comparing with state-of-the-an baseline methods.

---

References
[1] F. Ma and S. Karaman. Sparse-to-dense: Depth prediction from sparse depth samples and single image. In International Conference on Robotics and Automation, pages 4796-4803. IEEE, 2018.
[2] Y. Liao, L. Huang. Y. Wang, S. Kodagoda, Y. Yu, and Y. Liu. Parse geometry from a line: Monocular depth estimation with partial laser observation. In International Conference on Robotics and Automation, pages 5059-55000 5066. IEEE, 2017.
[3] N. Chodosh, C. Wang, and S. Lucey. Deep convolutional compressed sensing for lidar depth completion. In Asian Conference on Computer Vision, pages 499-513. Springer, 2018.
[4] A. Eldesokey, M. Felsberg, and F. S. Khan. Propagating confidences through cnns for sparse data regression, British Machine Vision Conference, 2018.
[5] X. Cheng, P. Wang, and R. Yang. Depth estimation via affinity learned with convolutional spatial propagation network. In European Conference on Computer Vision, pages 103-119, 2018.
[6] Z. Huang, J. Fan, S. Cheng, S. Yi, X. Wang, and H. Li. Hms-net: Hierarchical multi- scale sparsity-invarianl network for sparse depth completion. IEEE Transactions on Image Processing, 29:3429-3441, 2019.
[7] X. Cheng, Wang, and R. Yang. Learning depth with convolutional spatial propagation net- work. IEEE Transactions on Pattern Analysis and Machine Intelligence, 42(10):2361-2379, 2019.
[8] A. Eldesokey, M. Felsberg, and F. S. Khan. Confidence propagation through cnns for guided sparse depth regression. IEEE Transactions on Pattern Analysis and Machine Intelligence. 42 (10):2423 22222236,,,, 2019.
[9] S. Imran, Y. Long, X. Liu, and D. Morris. Depth coefficients for depth completion. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pages 13438-12444 IEEE, 2019.
[10] I. C. on Computer Vision. Depth completion from sparse lidar data with depth-normal con- straints, In International Conference on Computer Vision, pages 281 2811-2820, 2019.
[11] Yin, J. Qian, Cao, Held, and H. Choset. Fusionmapping: Learning depth prediction with monocular images and 2d laser scans. arXiv preprint arXiv: 1912.0006. 2019.
[12] V. Nekrasov, Dharmasiri, A. Spek, T. Drummond, C. Shen, and Reid. Real-time joint semantic segmentation and depth estimation using asymmetric annotations. In International Conference on Robotics and Automation, pages 7101-7107. IEEE, 2019.
[13] J. Qiu, Z. Cui, Y. Zhang, X. Zhang, S. Liu, B. Zeng, and M. Pollefeys. Deeplidar: Deep surface normal guided depth prediction for outdoor scene from sparse lidar data and single color image. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pages 3312-3322, 2019.
[14] Uhrig, N. Schneider, L. Schneider, U. Franke, T. Brox, and A. Geiger. Sparsity invariant cnns. In International Conference on 3D Vision, pages 11 11-20. IEFE, 2017.
[15] Y. Zhang and T. Funkhouser. Deep depth completion single rgb-d image. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 175-185, 2018.
[16] M. Hu, S. Wang, B. Li. Ning, Fan, and X. Gong. Penet: Towards precise and efficient image guided depth completion. arXiv preprint arXiv:2103.00788, 2021.
[17] D. Eigen, C. Puhrsch, and Fergus. Depth map prediction from single image using a multi-scale deep network. arXiv preprint arXiv: 1 406.2283. 2014.

---

[18] M. Jaritz, R. De Charette. E. Wirbel, X. Perrotton, and F. Nashashibi. Sparse and dense data with cnns: Depth completion and semantic segmentation. In International Conference on 3D Vision, pages 60. IEFE, 2018.
[19] B. Zoph, V. Vasudevan, J. Shlens, and Q. Le. Learning transferable architectures for scalable image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 8697-8710, 2018.
[20] D. Wofk, F. Ma, T.-J. Yang, S. Karaman, and V. Sze. Fastdepth: Fast monocular depth estima- tion on embedded systems. In International Conference on Robotics and Automation, pages 6101-6108, IEBE, 2019.
[21] C. Godard, O. Mac Aodha, M. Firman, and G. Brostow. Digging into self-supervised monoc- ular depth estimation. In International Conference on Computer Vision, pages 3828-30 -838,, 2019.
[22] H. Li, A. Gordon, H. Zhao, V. Casser, and Angelova. Unsupervised monocular depth learning in dynamic scenes. arXiv preprint arXiv:2010. 16404, 2020.
[23] F. Gao, J. Yu, H. Shen. Y. Wang, and H. Yang. Attentional separation-and aaae--ndd----gge aggregation network for self-supervised depth-pose learning in dynamic scenes. arXiv preprint 20.0969, 2020.
[24] O. Ronneberger, P. Fischer, and T. Brox. U-net: Convolutional networks for biomedi- cal image segmentation. In International Conference on Medical Image Computing and Computer-assisted Intervention, pages 234-241. Springer, 2015.
[25] Ma, G. Cavalheiro, and S. Karaman. Self-supervised sparse-to-dense Self-supervised depth completion from lidar and monocular camera. In IEEE International Conference on Roboties and Automation, pages 3288-3295. IERE, 2019.
[26] X. Cheng, P. Wang, C. Guan, and R. Yang. Cspn++: Learning context and resource aware convolutional spatial propagation networks for depth completion. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 10615-10622, 2020.
[27] A. G. Howard, M. Zhu, B. Chen, D. Kalenichenko, W. Wang, T. Weyand, M. Andreetto, and H. Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv: 704.04861, 2017.
[28] P. K. Nathan Silberman, Derek Hoiem and R. Fergus. Indoor segmentation and support infer- ence from rgbd images. In European Conference on Computer Vision, 2012.
[29] A. Geiger, P. Lenz, C. Stiller, and R. Urtasun. Vision meets roboties: The kitti dataset. The International Journal of Robotics Research, 32(11):2211-12) 2013.
[30] A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet classification with deep convolutional neural networks. Advances in Neural Information Processing Systems, 25:1097-1105, 2012.
[31] T. Chen, T. Moreau, Z. Jiang, L. Zheng, E. Yan, H. Shen, M. Cowan, L. Wang, Y. Hu, L. Ceze, C. Guestrin, and A. Krishnamurthy. TVM: An automated end-to-end optimizing compiler for deep learning. In 13th USENIX Symposium on Operating Systems Design and Implementation (OSDI 18), pages 578-594, Carlsbad, CA, 2018. ISBN 979-1-933133--8-3.
[32] Instruments. Ina226 high-side or low-side measurement, bi-directional current and power monitor with i2c compatible interface. INA226, pages 1-39, 2015.
10


Reviews:
[{"review_id": "t37guJWXBmv", "reviewer": "Reviewer_AwPn", "summary": "This paper proposes a depth estimation method with 2D lidar measurements and monocular RGB images as inputs, which could perform robustly with sparse and highly non-uniform depth measurements, and give reliable estimations when computation budget is limited.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The experimental results on NYU dataset clearly show the advantages of proposed method when working with very sparse depth measurements and low time budget over other state-of-the-art methods,  thus it could establish a new baseline and be a good reference work for relevant research in the future.\n\nThis work aims at training a model that provides robust predictions in anytime, hence samples time budget uniformly. I think it would also be interesting to study the actual distribution of interruption time in real applications and applies it to the sampling process, then the model may achieve better expected accuracy.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "This work targets a practical problem and provides a solution with better performance than other existing methods within the working situation, hence I think it has a high technical value. In addition, the proposed method has a clear and simple framework, thus it’s likely to be embedded to other applications conveniently.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "2: Unsure, difficult to assess.", "reviewer_expertise": "Fair: Some knowledge of the area"}, "extra_text": {"issues": "1. I have a question about the first way to optimize the objective in section 4.1: As the budget is sampled from the uniform distribution within (0, Bmax), the model should not be able to control it. But it’s said to minimize the budget here, then how does it work? I guess that the value of Bmax gradually decreases during training so that the model is trained to keep a similar performance within lower expected budget. Am I correct? \n\n2. Just a suggestion here: The formula of the loss function used in the optimization objective (equation (1)) is not mentioned. I think it would be helpful to the reader if you could briefly mention about it.\n"}}, {"review_id": "_fIYbGrR9lu", "reviewer": "Reviewer_rseL", "summary": "This work proposes an approach for anytime monocular depth estimation on mobile devices when a structured sparse set of measurements in the form of a 2D lidar is available. The authors propose to encode LiDAR scanline measurements using a neural network and splat them to image features that are encoded using a light-weight image network. ", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n- Anytime estimation in this setting (e.g. with additional external measurements from LiDAR) is to the best of my knowledge novel. \n\n- Important and undeserved topic. There are only few works in this general direction despite its obvious usefulness for robotics applications.\n\n- Good results in comparison to other real-time approaches.\n\nWeaknesses:\n\nClarity is an issue throughout the manuscript:\n\n- Section 3.2 states a relatively simple concepts in a bit heavy-handed terms because of the extensive use of mathematical notation. I'd recommend to make this a bit more accessible by removing jargon and explaining with words. \n\n- Section 4.1: similar consideration hold about overly complicated notation, but here I'm not sure if I completely understand the motivation here. For point one: how can you guarantee even in expectation the inequality if the losses are only required to be equal? What happens if the loss for \\hat f_i is actully slightly larger on |\\hat P_i} is only slightly smaller that |P_i|?\n\n- Paper organization: Section 4.2 talks about network architecture followed by an own section 4.3 that is actually titled \"network architecture\" but then actually goes on to talk about computational properties. This is confusing. It is not clear what \"encoder optimization\" means. What is optimization referring to here? The ECU is used as a component and how it qualitatively functions is mentioned already in Section 4.2, but it only introduced in 4.3.2. \n\n- Section 4.3.2 is very hard to understand. What alignment is necessary here? Why is this necessary? How big are the regions? What is the compensation value? It seems that this is just about upsampling the scan line to the full image by extrapolating the scan line in vertical dimension. Why is this so complicated?\n\n- Is a uniform distribution between 0 and B_max well justified? In practice it seems that this would more like by an interval [B_min, B_max] or even a normal distribution.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I really like the direction that this paper is taking. Unfortunately, the paper in its current form is extremely hard to read and in its current form not fit for publication. Technical aspects are consequently hard to evaluate in detail, but I believe that the paper in principle is on a good direction. I'm willing to significantly improve my score if the revision appropriately improves clarity. \n\n-- Post rebuttal --\n\nThe authors answered my questions and revised the manuscript. I'm consequently raising my score.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "2: Unsure, difficult to assess.", "reviewer_expertise": "Good: General knowledge of the area"}, "extra_text": {"issues": "Minor issues\n\n- l. 121 abbreviation \"r.v.\" is never introduced.\n\n- The abbreviation PC is used in almost every sentence in some paragraphs which make these parts dull to read.\n\nMissing citation: \n\n[A] also proposes an efficient depth estimation network that outperforms [18].\n\n[A] Nekrasov et al. Real-time joint semantic segmentation and depth estimation using asymmetric annotation. ICRA 2019"}}, {"review_id": "Bq3mPaf3ztN", "reviewer": "Reviewer_2XnV", "summary": "The paper addresses the problem of depth estimation using a monocular camera and a 2D LiDAR sensors. The paper proposes an anytime method based on a network architecture that is composed of two dedicated encoders map to extract features from the input of each sensor, a unit to fuse such features, and a decoder to get the final depth map. The proposed method accounts for time/energy budget available to potentially stop the inference and get the corresponding result. Experimental results obtained on public datasets are shown to validate the proposed method.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Based on existing networks, the paper presents an interesting idea about stopping the inference and explains the motivation behind the choices that resulted in a much smaller network compared to other methods, displaying comparable or better performance in some cases. The paper is also generally well written.\n\nThere are two main comments:\n- the paper could provide better examples where this stopping is useful for energy purpose: a self-driving car or any moving robot in general typically consumes much more energy from the actuators rather than from the computer. It would be useful to quantify the energy saving.\n- the budget model (constant) is quite simplistic and might result in prematurely stopping the inference, as the CPU load might be different at different times. A discussion on a more realistic model would be useful.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "While the paper could provide a better motivation and discuss a more realistic budget model, the paper presents an interesting anytime method for depth estimation targeted for low-powered systems.\n\n", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "Good: General knowledge of the area"}, "extra_text": {"issues": "The issues that could be addressed to revise the paper are reported in the \"Strengths And Weaknesses\" textbox.\n\nOther specific comments:\n- in practice, it might be difficult to get the line of points at the center of the ground truth depth map, unless the camera and LiDAR are colocated. Thus, taking the line above or below the center would be more realistic.\n- Besides just reporting the data, it will be useful to comment on when other methods perform better than the proposed one, e.g., in Table 3. \n\nMinor typos: \n- \"equiped\" -> \"equipped\"\n- \"an uniform\" -> \"a uniform\" (because of the consonantal sound)"}}, {"review_id": "95oBZI0kmG7", "reviewer": "Reviewer_HGbf", "summary": "This paper presents  RGB-D based, computationally efficient, encoder-decoder style depth estimation network with a focus on low power, frequently halting applications.   Unlike other RGB-D fusion architectures, to mimic the limitations existed on small devices, authors use RGB and 2D point cloud as model's input instead of 3D point clouds, and propose an Error Correction Unit, ECU, to algin 2D point cloud and RGB image features. ECU achieves that by extracting point cloud's neighbors in image features and downsizing it to match point cloud feature dimension, and the combined feature is upscaled back to image feature dimension.  Additionally, they incorporate five depth estimation outputs along the computation pipeline in order to achieve anytime depth estimation, where predictions with different levels of detail are available regardless of the halting point of the network.  Authors propose a new metric, anytime loss, which is a weighted sum of loss for each depth output based on everyone's computational time. ", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The idea of anytime depth estimator is novel and useful practically for time-sensitive and low power edge devices. \n2. Error correction unit seems like an interesting approach to fuse image and 2D LiDAR features with different dimensions, existing methods commonly use interpolations to expand LiDAR features to image dimension.\n3. Comparing results of common models and authors' model on small devices like Jetson and Raspberry Pi with limited time and energy budget is useful for future applications on such edge devices. As many publications focus on SOTA performance on powerful devices, obtaining results on weak devices can guide the community to pay attention to efficiency and ease of deployment of their networks.\n\nWeaknesses:\n1. Since all but the first nearest interpolation predictions require the encoder section of the network to be completed, and the first prediction is significantly worse than others, this paper's claim about anytime estimations still requires at least half of the total computation time to yield useful predictions. (In Table 1 path1 takes 6.4ms whereas path4 takes 12.2ms). Would be nice to alter the network architecture to allow it to bypass some layers of encoders to achieve a lower minimum time requirement.\n2. Another question I have about architecture is that although authors claim that the network is based on U-net and uses first three conv layers of MobileNetV1 as encoder. Yet, MobileNectV1's first three layers only reduce image dimension by half, and as 1x1 conv layer is used to generate full-size depth estimation. I am not sure if the network downsamples and upsamples features similar to U-net. \n3. I think this paper doesn't demonstrate the effectiveness of ECU module, as from the results, Parse, which uses 2D LiDAR by interpolation, performs similarly compared to the proposed network. Would be nice to have a comparison between proposed network with and without ECU module.\n4. I think \"FusionMapping: Learning Depth Prediction with Monocular Images and 2D Laser Scans\" should be included as baseline as it shares a similar architecture with the proposed network and uses 2D LiDAR as input as well.\n5. Finally, I have a different opinion about the proposed anytime estimator approach to handle limited time and energy budget for edge devices. The paper considers the halting caused by limited budgets to be a uniformly distributed random variable from 0 to B_mas, and designs the model to give prediction whenever halting happens. However, I would argue that the capturing and processing time for RGBD image can be considered as a constant. Thus, instead of designing a model to perform well under different budgets, constructing a model to fit the given budget seems to be a better option.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Overall, this paper shows a novel approach for low power edge device machine learning by handling limited time budgets by anytime depth estimation. The proposed architecture is similar to other RGBD depth estimators with encoder-decoder backbone and point cloud feature fusion, yet the authors introduce multiple branches of prediction between decoder layers to obtain depth estimation with different levels of qualities. Also, a novel anytime loss is introduced to weigh each output's loss by its computational time. Another novel part is the error correction module for RGB and point cloud feature fusion. The paper demonstrates the network's low latency and good performance comparable to existing methods on low power Jetson and Raspberry Pi. However, the newly introduced ECU layer lacks experimental results to demonstrate its effectiveness. And of the claim of anytime prediction, the model design cannot give outputs if halt happens during encoding. Additionally, random halting doesn't match with the situation on low power devices, instead, a fixed time budget seems to be common, which wouldn't have random halting problem and allows a purposely designed model to fully utilize the time budget. ", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "2: Unsure, difficult to assess.", "reviewer_expertise": "Fair: Some knowledge of the area"}, "extra_text": {"issues": "1. Compare the proposed network with and without ECU to demonstrate its effectiveness.\n2. add FusionMapping as a baseline \n3. try have halt in encoder step and provide the anytime loss for that.\n4. clarify the architecture and if not doing downsampling and upsampling, justify the design choice."}}, {"review_id": "64Z0mmq7GRe", "reviewer": "Reviewer_Y4hj", "summary": "The paper presents a mono depth completion method based on a single 2D LIDAR scan. The whole model is designed to have a limited computational budget and to exit the inference at different stages depending on the currently available power or computational budget. It shows outstanding results even though the amount of weights and the amount of flops are drastically reduced. ", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n* Nice and clear presentation of the approach\n* Well written paper\n* In depth evaluation of the method and comparison on two different relevant dataset\n* Introduction on an error-correction unit (ECU), which fuses the 2D features from a mobilenet with the features extracted from a 1D conv-net, by extracting the local neighborhood in the spatial image feature domain and concatenating it with the results from the 1D LIDAR scan features. \n* Four way exit approach, which allows to get results after only half the time of execution of the full pipeline\n* Impressive results on NYU Depth v2, where after 6.4ms the method performs already 5% better than FastDepth [18], which needs in comparison 17.1ms\n\nWeaknesses:\n* Figure 5 should include the 33 and 50 ms mark, to better compare it with Table 2. \n* Line 267: should better explain that the Raspberry Pi 4 takes 59 ms to get to the first path, which means that is not fast enough for a fair comparison here\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The paper was well written and fun to read. I can strongly recommend accepting this paper to CORL. I have a few pointers, which should be improved in a final version, but nothing which should prevent this paper from getting accepted.\n\n", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "Good: General knowledge of the area"}, "extra_text": {"issues": "* Figure 5 should include the 33 and 50 ms mark, to better compare it with Table 2. \n* Line 267: should better explain that the Raspberry Pi 4 takes 59 ms to get to the first path, which means that is not fast enough for a fair comparison here"}}]

Author rebuttal:

Meta-review: The paper presents a mono depth completion method based on a single 2D LIDAR scan.
The paper proposes an anytime method based on a network architecture that is composed of two dedicated encoders map to extract features from the input of each sensor, a unit to fuse such features, and a decoder to get the final depth map. Overall, this paper shows a novel approach for low power edge device machine learning by handling limited time budgets by anytime depth estimation.

All reviewers but one are positive and find the paper addressing an important question and well written/executed with interesting results. 
- Clarity on how the budget is used: constant or adaptive? If latter how, and what are the implications of the assumptions on uniform sampling vs say exponential or gaussian distributions. 
- Choice and justification for architecture (as pointed out by Reviewer HGbf)

Overall the rebuttal was very useful in clarifying these questions. The authors should ensure that most points from the rebuttal are included in the main paper and if not then should be added to the appendix as FAQs. This will improve comprehension and avoid similar confusion among the readers. 
Decision: Accept (Poster)