Title: RETHINKING SELF-DRIVING: MULTI-TASK KNOWL-EDGE FOR BETTER GENERALIZATION AND ACCIDENT EXPLANATION ABILITY

Abstract: Current end-to-end deep learning driving models have two problems: (1) Poor generalization ability of unobserved driving environment when diversity of training driving dataset is limited (2) Lack of accident explanation ability when driving models don't work as expected. To tackle these two problems, rooted on the believe that knowledge of associated easy task is benificial for addressing difficult task, we proposed a new driving model which is composed of perception module for see and think and driving module for behave, and trained it with multi-task perception-related basic knowledge and driving knowledge stepwisely. Specifically segmentation map and depth map (pixel level understanding of images) were considered as what & where and how far knowledge for tackling easier drivingrelated perception problems before generating final control commands for difficult driving task. The results of experiments demonstrated the effectiveness of multitask perception knowledge for better generalization and accident explanation ability. With our method the average sucess rate of finishing most difficult navigation tasks in untrained city of CoRL test surpassed current benchmark method for 15 percent in trained weather and 20 percent in untrained weathers.

1. INTRODUCTION:

Observing progressive improvement in various fields of pattern recognition with end-to-end deep learning based methods (Krizhevsky et al., 2012; Girshick, 2015) , self-driving researchers try to revolutionize autonomous car field with the help of end-to-end deep learning techniques (Bojarski et al., 2016b; Chen et al., 2015; Codevilla et al., 2017) . Impressive results have been acquired by mapping camera images directly to driving control commands (Bojarski et al., 2016b) with simple structure similar to ones for image classfication task (Simonyan & Zisserman, 2014) . Further researches were conducted to improve the performance of deep learning based autonomous driving system, for example, Conditional Imitation Learning (Codevilla et al., 2017) approach has been proposed to solve the ambigious action problem.

However, two crutial problems failed to be spotted: (1) Poor generalization ability of unobserved driving environment given limited diversity of training scenerios. For example, though Dosovitskiy et al. (2017) addressed the driving direction selection problem, it showed poor generalization ability in unseen test town which has different map and building structure than training town's. This generalization problem is extremely important since collected driving dataset always has limitation of diversity (2) Current end-to-end autonomous approaches lack of accident explanation ability when these models behave unexpectedly. Although saliency map based visualization methods (Smilkov et al., 2017; Sundararajan et al., 2017; Springenberg et al., 2014; Bojarski et al., 2016a) have been proposed to dig into the 'black box', the only information these methods could bring is the possible attention of the model instead of the perception process of the model.

We proposed a new driving approach to solve the two aforementioned problems by using multi-task basic perception knowledge. We argue that when end-to-end model is trained to address a specific difficult task, it's better to train the model with some basic knowledge to solve relevant easier tasks before (Pan et al., 2010) . An analogy for this can be observed when human beings learn a difficult knowledge. For example, to solve a complex integration problem, compared with students without basic math knowledge, students who know about basic knowledge of math are able to learn the core of intergration more quickly and solve other similar integration problems instead of memorizing the solution of the specific problem.

Our proposed model consists of two modules: perception module and driving module as in Fig. 1 . The perception module is used for learning easier driving-related perception knowledge, which we refer as ability of pixel level understanding of input including what & where and how far knowledge. We trained perception module with segmentation map and depth map first, while the former serves as what & where knowledge and the latter serves as how far knowledge. By visualizing inferenced segmentation and depth results whether perception process works well or not could be inferred. After the perception module was trained to have ability of pixel level understanding of its image input, we freezed the perception module weights and trained driving module with driving dataset. This decomposition of end-to-end driving network strucuture is considered to be mediated perception approach (Ullman, 1980) . With our proposed driving structure and stepwise training strategy, the generalization and accident explanation problems were addressed to a certain extent.

2. RELATED WORK:

Depending on whether mediated perception knowledge are generated, self-driving models are categorized into mediated perception approach (Ullman, 1980) and behavior reflex approach.

For mediated perception approaches, there are several well-behaved deep learning methods, for example, Deep-Driving method (Chen et al., 2015) fisrtly converts input RGB images to some key perception indicators related to final driving controls. They designed a very simple driving controller based on predicted perception indicators. Problem of this approach is that key perception indicators have limitation of describing unseen scenerios and are difficult to collect in reality. Except for inferencing for final driving controls, there are approaches which focus on inferencing intermediate description of driving situation only. For separate scene understanding task, car detection (Lenz et al., 2011) and lane detection (Aly, 2008) are two main topics in this area.

Instead of inferencing one perception task at most, multi-task learning method aims at tackling several relevant tasks simultaneously. Teichmann et al. (2016) uses input image to solve both object detection and road segmentation tasks. Branched E-Net (Neven et al., 2017) not only infers for segmentation map, but also depth map of current driving scenarios. These multi-task learning methods shows better result when sharing the encoder of different perception tasks together, but they haven't really tried to make the car drive either in simulator or reality.

As for behavior reflex approach which is also called 'end-to-end learning', NVIDIA firstly proposed a model for mapping input image pixels directly to final driving control output(steer only) (Bojarski et al., 2016b) . Some other approaches further atempted to create more robust models, for example, long short-term memory (LSTM) was utilized to make driving models store a memory of past (Chi & Mu, 2017) .

One problem is that aforementioned methods were tested in dissimlar driving scenerios using different driving dataset, thus it's hard to determine if model itself is the source of the better driving behavior instead of effectiveness of data (Sun et al., 2017) . Codevilla et al. (2017) was tested in a public urban driving simulator Dosovitskiy et al. (2017) and sucessed to tackle the ambigous action problem which refers as optimal driving action can't be inferred from perceptual input alone. Benefit from CoRL test in Dosovitskiy et al. (2017) , fair comparision could be conducted using same driving dataset. Codevilla et al. (2017) showed limitation of generalization ability problem in test town different from train town (Dosovitskiy et al., 2017) as in CoRL test training dataset could be only collected from single train town.

When the end-to-end driving method behaves badly and causes accidents, accident explanation ability is required. Though saliency-map based visualization methods (Bojarski et al., 2016a; Smilkov et al., 2017) help understand the influence of input on final driving control, it's extremely hard to derive which module of the model fails when driving problems happen -If the model percepts incorrectly or the driving inference processes wrongly based on good perception information. Driving system was enabled to give quantitative explanation by visualizing inferenced multi-task basic knowledge to solve this problem.

3. FRAMEWORK OF PROPOSED SYSTEM:

Basic strucure of the proposed model is shown in Fig. 1 . The proposed model has two parts: (1) Multi-task basic knowledge perception module (2) Driving decision branch module. The perception module is used to percept the world by inferencing dpeth map and segmentation map, which is composed of one shared encoder and two decoders for two different basic perception knowledge:

(1) Segmentation decoder for generating 'what & where' information by predicting segmentation maps; (2) Depth decoder for predicting 'how far' the objects in vision are by inferencing depth maps. The perception module is aimed at extracting encoded feature map containing pixel level understanding information for driving module and qualitative explanation when proposed model doesn't work as expected by visualizing the predicted segmentation and depth maps to determine if the driving problem is caused by percept process or driving process.

The driving module enbales the model to generate driving decisions for different direction following guidances. We categorized the real world driving guidance into four types: (1) Following lane (2) Turning left (3) Going straight (4) Turning right as done in Codevilla et al. (2017) . For each driving guidance direction, there is a driving branch(which predicts the value of driving controls) corresponding to it, therefore there are four driving guidance branches totally. The output of second last layer in perception module is inputted to the driving module, therefore the training of which could benefit from the multi-knowledge extracted by the perception module. Instead of linear layers, convolution layers are utilized for inferencing final driving controls for each direction, which helps keeping the spatial relation of information and reducing number of parameters as non-negligible quantity of direction branches.

3.1. MULTI-TASK BASIC KNOWLEDGE PERCEPTION MODULE:

The perception module is built with residual block proposed in (He et al., 2016) which solves gradient vanishing and 'degradation problem', and it has a structure similar to Segnet (Badrinarayanan et al., 2015) prosposed for efficient image segmentation task. Huge difference is that in our proposed method there are two different decoders for inferencing both segmentation and depth maps simultaneously instead of segmentation map only. Besides, we constraint the total strides in encoder to 8 for keeping resolution of feature map, as large total stride has negative influence on feature map size reconstuction. Hybrid Dilated Convolution Wang et al. (2017) is adapted as last part of the encoder as it enlarges the receptive field and avoids theoretical issue of gridding problem. Groupout (Park) is also adapted to avoid overfitting problem in the convolution network.

3.2. DRIVING DECISION BRANCH MODULE:

The driving module is built with residual block and has a general form as Codevilla et al. (2017) in last output layer for several direction outputs. It is all based on convolutional layers in order to keep the spatial information and reduce parameters motivated by Springenberg et al. (2014) . Four different high level driving guidance such as "turning right" are utilized for selecting which direction branch's output is supposed to be considered as final driving outputs. Driving outputs contain steering and acceleration/brake, both of them range from -1 to 1. Since there are 4 output branches corresponding to 4 high level driving guidances, 8 feature map size convolution kernels are set in the last layer for output scalar value, in which each two are regarded as driving controls for one driving guidance. To determine the limitation of RGB image, no other information such as current speed or steering angle were used as input. Instead we atempted to predict the current speed based on current RGB image to keep the driving smoothly as done in (Codevilla et al., 2017) . The input of the driving module is not from the last layer's output of the encoder part in the perception module, but the second last layer's output of the encoder part due to empirically selection for best generalization.

4. EXPERIMENTS:



4.1. SYSTEM SETUP:

The training dataset is collected in CARLA simulator (Dosovitskiy et al., 2017) . CARLA simulator is a self-driving simulator developed by Intel Co. for collecting self-driving related information and evaluating driving model with a standard testing environment named CoRL test. CoRL test is composed of 4 tasks of increasing difficulty: (1) Straight: the goal is straight ahead of the starting position. (2) One turn: getting to the goal takes one turn, left or right. (3) Navigation: navigation with an arbitrary number of turns. (4) Navigation with dynamic obstacles: same as previous task, but with other vehicles and pedestrians (Dosovitskiy et al., 2017) .

The main metric for quantitatively evaluating is the average success rate of finishing seperate tasks in CoRL test. CoRL test contains tests both in trained town and untrained town under both trained and untrained weathers. Test trained town and untrained town are constructed with different maps and different building texture.

4.2. DATASET:

Dataset for training our model could be categorized into 2 items: (1) Perception module training dataset (2) Driving module training dataset. For perception module, we trained it with 35,000 pairs of RGB images, segmentation and depth maps and evaluated with 5,000 pairs. As for driving module, we trained with 455,000 dataset, and evaluated on 62,000 evaluation dataset. Before training our proposed model, two vital data processing methods were used: balancing dataset and data augmentation. For fair comparison, we use same driving dataset published by Conditional Imitation Learning (Codevilla et al., 2017 ) except that we collected extra segmentation and depth maps in train town for training our proposed perception module.

4.2.1. DATA BALANCING:

Dataset balancing contributed to better generalization of both perception module and driving module in our experiments as it enables each mini-batch to be a microcosm of the whole dataset. For perception module, dataset were balanced to ensure that each mini-batch contains all different training weathers and an equal amount of going straight and turning situations. For driving module, we balance each training mini-batch to ensure equally distribution of different driving direction guidance, and reorganized large steer(absolute value larger than 0.4) data accounts for 1/3 in each mini-batch, brake situation data acounts for 1/3, noise steer situation for 1/10.

4.2.2. DATA AUGMENTATION:

We add guassian noise, coarse dropout, contrast normalization, Guassian blur to both training dataset for perception and driving module for enlarging training dataset distribution.

4.3. TRAINING DETAILS:

We trained the whole system using a step-wise training method which is firstly we trained the perception module with multi-task basic perception knowledge, then we freezed the weights of perception module and train driving module with driving dataset. For training the perception module, we used mini-batch size 24 and set ratio of segmentation loss and depth loss to be 1.5:1. Softmax categorical crossentropy is used for segmentation loss, binary crossentropy is used for depth loss. Adam of 0.001, which is multiplied by a factor of 0.2 of previous learning rate if validation loss does't drop for 1 epoch is used as optmizer. L2 weight decay and early stopping are also used for avoid overfitting. As for training the driving module, we consider MSE loss and use Adam with starting learning rate of 0.002 which exponentially decay of 0.9 every epoch. Early stopping and L2 decay are used for regularization.

5. EXPERIMENTS & RESULTS:



5.1. GENERALIZATION ABILITY TEST:

We compare the results of driving performance between our proposal and other methods tested in CoRL test via success rate of finishing each task. The details of results are shown in We could also visualize the perception process when the model works. One example of test in untrained town and untrained weather is shown in Fig. 2 .

Figure 2 : Screenshots of captured single RGB image and our proposal's inference results which consists of predicted segmentation map and depth map during test in untrained town under untrained weather. Obviously from the inferenced segmentation and depth maps we get information that the driving model knows a car is passing by the left side.

5.2. ORIGIN OF BETTER GERNERALIZATION ABILITY:

Since we observed our training model has better generalization ability in unseen town comparing with other methods when almost the same driving dataset were used to train (except that we collected extra depth maps and segmentation maps in same training environments), we want to investigate the origin of the better generalization ability. There are two possible reasons why our training model has better generalization ability in unseen town: (1) Basic knowledge (segmentation map and depth map) (2) Network structure. Therefore we conduct experiments by comparing performance of two methods:

• Our original proposal: Firstly train perception module with basic knowledge, after training perception module, freeze its weights and train driving module with driving dataset • Compared method: Train the encoder of perception module and driving module together with driving dataset. No basic perception knowledge is used for training model.

Since tests in CoRL cost much time, we limited our evaluation to the most difficult untrained town under untrained weathers test. Results are shown in Table . 2. From the results it's obvious that multibasic knowledge we use in the training phase is the origin of our proposal's good generalization ability of untrained town instead of the network structure. Moreover, the network structure could be improved to achieve better performance in the furture.

5.3. QUALITATIVE CAUSE EXPLANATION ABILITY OF DRIVING PROBLEMS:

Besides basic knowledge leads to better generalization ability, it could also be used to give a qualitative explanation of driving problems. Basic knowledge of segmentation map and depth map are output from the perception module during test phase, therefore how the driving module percepts the current scenario could be known by simply visualizing the outputs of segmentation and depth from perception module. Depending on the predicted pixel understanding of the situation, cause of driving problem could be inferred.

One example is shown in Fig. 3 . For a failed straight task in untrained town under untrained weather soft rain sunset as the driving model failed to move forward, we visualized outputs of segmentation and depth maps predicted by perception module. It's obvious that this failure case is caused by the perception module since the model falsely percepted that there is a car in front of it and in order to avoid collision it did't start. There is no car actually thus the perception module made false judgement. However, what's interesting is that sometimes it fools readers to think that there is a car in Fig. 3 because of sun ray reflection on the wet road and the perception module has the similar understanding as these readers. Therefore in some aspects the perception module makes the right judgement instead of wrong's. For traditional end-to-end learning driving methods (Bojarski et al., 2016b) it's impossible to reason as they don't focus on the cause explanation ability which is of great importance for practice use of deep learning driving models.

Figure 3 : Failed straight task in untrained town under untrained soft rain sunset weather. From predicted segmentation and depth maps, we know that the driving model thinks that there is a car in front of it but actually there isn't any car in front of it.

5.4. FINE-TUNE TEST:

Fine -tuneYosinski et al. (2014) , which refers to use other well-trained models weights on different target-related dataset as initial weights for training with target dataset instead of using weights initializing methods (Glorot & Bengio, 2010; He et al., 2015; LeCun et al., 2012) , is a common trick used in deep learning since empirically it could leads to better generalization on new target dataset.

Here in our specific case we refer fine-tune method to be after training perception module we train the weights of the encoder of the perception module as well instead of freezing these weights. In Table . 3 we compare the performance of fine-tune method and our original proposed method. In this comparison we achieved counter-intuition results: after fine-tune the weights of the perception module the driving model achieved worse results than original method which freeze the weights of perception module when training the driving module.

One possible reason is that the generalization ability lies in the perception module instead of the driving module, therefore when we train the perception module again with driving dataset, the ability of generating compressed multi-knowdege information is destoryed. As the fine-tune model couldn't benefit from the multi-task knowledge anymore, it failed to produce the same generalization ability as the original proposal did.

Furthermore we conduct experiment on visualizing one direction of loss surface by projecting the loss surface to 2 dimension (Goodfellow & Vinyals, 2014) to investigate some qualitative explanation for this comparison result. x axis corresponds to linear interpolation of the weights of original proposed method and weights of compared fine-tuned method after training. Formulation of calculating the weights in this projection direction is Equation. 1.

EQUATION

α is linear interpolation ratio, x f intune and x rgb0 are trained weights of fine-tune method and original proposal method. f (x) is loss function of the whole model while input is considered as different model weights. We draw out the projected loss surface as Fig. 4 by sampling from the interpolation weigthts. From Fig. 4 we can get one possible qualitative reason for worse behavior of fine-tune method from a loss surface perspective: Model weight got by using fine-tune method is stuck in a super flat surface, while model weights of original proposed method successfully finds a local minimum.

6. CONCLUSION:

In this paper we propose a new driving system for better generalization and accident explanation ability by enabling it to do simpler driving-related perception task before generating commands for diffult driving task. Through multiple experiments we empirically proved the effectiveness of the multi basic perception knowledge for better generalization ability of unobserved town when diversity of training dataset is limited. Besides our proposed model has self-explanation ability by visualizing the predicted segmentation and depth maps from the perception module to determine the cause of driving problems when they happen. One interesting result we acquired by comparing different train strategies is that the generalization ability of driving origins from basic knowledge and lies in weights of the perception module which should not be modified during training with driving dataset. We hope our work could movitivate other researches to use multi-task target related perception knowledge for better performance in robot learning. In future we will investigate more effective network structures. 

:



Figures and Tables:

Figure 1: Basic structure of proposed self-driving system. The components which lie in the red bounding box are refered as parts of perception module. The components which lie in the blue bounding box forms the driving module. In order to see the limiation of RGB image, only current RGB image and driving guidance are used as inputs seperatedly for perception module and driving module.

Figure4: One perceptive of loss surface by linear interpolation of original proposed method weights and fine-tune method weights. Blue line refers to test loss, red line refers to train loss. From the visualization perspective it's possibly that finetune method weights are stuck in a flat surface while the original proposed weights sucessfully find a local minimum.

Quantitive evaluation of methods in the goal-directed navigation tasks in CoRL test. The table reports the percentage of success rate of finishing corresponding task in different condition. Higher means better performance.

Table content: <table><tr><td/><td/><td colspan="2">Training conditions</td><td/><td>New town</td><td/><td/><td colspan="2">New weather</td><td colspan="3">New town&amp;weather</td></tr><tr><td>Task</td><td colspan="12">MP IL RL OURS MP IL RL OURS MP IL RL OURS MP IL RL OURS</td></tr><tr><td>Straight</td><td>98</td><td colspan="2">95 89 98</td><td>92</td><td colspan="2">97 74 100</td><td colspan="3">100 98 86 100</td><td>50</td><td colspan="2">80 68 96</td></tr><tr><td>One turn</td><td>82</td><td colspan="2">89 34 87</td><td>61</td><td colspan="2">59 12 81</td><td>95</td><td colspan="2">90 16 88</td><td>50</td><td colspan="2">48 20 82</td></tr><tr><td>Navigation</td><td>80</td><td colspan="2">86 14 81</td><td>24</td><td>40 3</td><td>72</td><td>94</td><td>84 2</td><td>88</td><td>47</td><td>44 6</td><td>78</td></tr><tr><td colspan="2">Nav. dynamic 77</td><td>83 7</td><td>81</td><td>24</td><td>38 2</td><td>53</td><td>89</td><td>82 2</td><td>80</td><td>44</td><td>42 4</td><td>62</td></tr></table>

Quantitive evaluation of original proposal which uses segmentataion and depth maps in training phase and compared method which doesn't use. Results shows that when multi-knowldge is not used for training, the success rate drops hugely in the testing town under untained weathers. It indicates that multi-basic knowledge is the main origin of the good generalization ability in our proposal.

Table content: <table><tr><td/><td colspan="2">New town&amp;weather test</td></tr><tr><td>Task</td><td colspan="2">COMPARED ORIGINAL</td></tr><tr><td>Straight</td><td>91</td><td>96</td></tr><tr><td>One turn</td><td>52</td><td>82</td></tr><tr><td>Navigation</td><td>20</td><td>78</td></tr><tr><td>Nav. dynamic</td><td>16</td><td>62</td></tr></table>

Quantitive evaluation of original proposal and fine-tune method which doens't freeze the perception module's weights during training the driving module. Results shows that the success rate of the fine-tune method drops dramaticaly in the testing town under untained weathers.

Table content: <table><tr><td/><td colspan="2">New town&amp;weather test</td></tr><tr><td>Task</td><td colspan="2">FINETUNE ORIGINAL</td></tr><tr><td>Straight</td><td>88</td><td>96</td></tr><tr><td>One turn</td><td>59</td><td>82</td></tr><tr><td>Navigation</td><td>42</td><td>78</td></tr><tr><td>Nav. dynamic</td><td>33</td><td>62</td></tr></table>

Peer review (iclr2019):
 # Summary This submission proposes a multi-task convolutional neural network architecture for end-to-end driving (going from an RGB image to controls) evaluated using the CARLA open source simulator. The architecture consists of an encoder and three decoders on top: two for perception (depth prediction and semantic segmentation), and one for driving controls prediction. The network is trained in a two-step supervised fashion: first training the encoder and perception decoders (using depth and semantic segmentation ground truth), second freezing the encoder and training the driving module (imitation learning on demonstrations). The network is evaluated on the standard CARLA benchmark showing better generalization performance in new driving conditions (town and weather) compared to the CARLA baselines (modular pipeline, imitation learning, RL). Qualitative results also show that failure modes are easier to interpret by looking at predicted depth maps and semantic segmentation results. # Strengths Simplicity of the approach: the overall architecture described above is simple (cf. Figure 1), combining the benefits of the modular and end-to-end approaches into a feed-forward CNN. The aforementioned two-stage learning algorithm is also explained clearly. Predicted depth maps and semantic segmentation results are indeed more interpretable than attention maps (as traditionally used in end-to-end driving). Evaluation of the driving policy: the evaluation is done with actual navigation tasks using the CARLA (CoRL*18) benchmark, instead of just off-line behavior cloning accuracy (often used in end-to-end driving papers, easier to overfit to, not guaranteed to transfer to actual driving). Simple ablative analysis: Table 2 quantifies the generalization performance benefits of pretraining and freezing the encoder on perception tasks (esp. going from 16% to 62% of completed episodes in the new town and weather dynamic navigation scenario). # Weaknesses ## Writing I have to start with the most obvious one. The paper is littered with typos and grammatical errors (way too many to list). For instance, the usage of "the" and "a" is almost non-existent. Overall, the paper is really hard to read and needs a thorough pass of proof-reading and editing. Also, please remove the acknowledgments section: I think it is borderline breaking the double-blind submission policy (I don*t know these persons, but if I did that would be a breach of ICLR submission policy). Furthermore, I think its contents are not very professional for a submission at a top international academic venue, but that is just my opinion. ## Novelty This is the main weakness for me. The architecture is very close to at least the following works: - Xu, H., Gao, Y., Yu, F. and Darrell, T., End-to-end learning of driving models from large-scale video datasets (CVPR*17): this reference is missing from the paper, whereas it is very closely related, as it also shows the benefit of a segmentation decoder on top of a shared encoder for end-to-end driving (calling it privileged training); - Codevilla et al*s Conditional Imitation Learning (ICRA*18): the only novelty in the current submission w.r.t. CIL is the addition of the depth and segmentation decoders; - Müller, M., Dosovitskiy, A., Ghanem, B., & Koltun, V., Driving Policy Transfer via Modularity and Abstraction (CoRL*18): the architecture also uses a shared perception module and segmentation (although in a mediated way instead of auxiliary task) to show better generalization performance (including from sim to real). Additional missing related works include: - Kim, J. and Canny, J.F., Interpretable Learning for Self-Driving Cars by Visualizing Causal Attention (ICCV*17): uses post-hoc attention interpretation of "black box" end-to-end networks; - Sauer, A., Savinov, N. and Geiger, A., Conditional Affordance Learning for Driving in Urban Environments (CoRL*18): also uses a perception module in the middle of the CIL network showing better generalization performance in CARLA (although a bit lower than the results in the current submission). - Pomerleau, D.A., Alvinn: An autonomous land vehicle in a neural network (NIPS*89): the landmark paper for end-to-end driving with neural networks! ## Insights / significance In light of the aforementioned prior art, I believe the claims are correct but already reported in other publications in the community (cf. references above). In particular, the proposed approach uses a lot more strongly labeled data (depth and semantic segmentation supervision in a dataset of 40,000 images) than the competing approaches mentioned above. For instance, the modular pipeline in the original CARLA paper uses only 2,500 labeled images, and I am sure its performance would be vastly improved with 40,000 images, but this is not evaluated, hence the comparison in Table 1 being unfair in my opinion. This matters because the encoder in the proposed method is frozen after training on the perception tasks, and the main point of the experiments is to convince that it results in a great (fixed) intermediate representation, which is in line with the aforementioned works doing mediated perception for driving. The fine-tuning experiments are also confirming what is know in the litterature, namely that simple fine-tuning can lead to catastrophic forgetting (Table 3). Finally, the qualitative evaluation of failure cases (5.3) leads to a trivial conclusion: a modular approach is indeed more interpretable than an end-to-end one. This is actually by design and the main advocated benefit of modular approaches: failure in the downstream perception module yields failure in the upstream driving module that builds on top of it. As the perception module is, by design, outputting a human interpretable representation (e.g., a semantic segmentation map), then this leads to better interpretation overall. ## Reproducibility There are not enough details in section 3.1 about the deep net architecture to enable re-implementation ("structure similar to SegNet", no detailed description of the number of layers, non-linearities, number of channels, etc). Will the authors release the perception training dataset collected in CARLA described in Section 4.2? # Recommendation Although the results of the proposed multi-task network on the CARLA driving benchmark are good, it is probably due to using almost two orders of magnitude more labeled data for semantic segmentation and depth prediction than prior works (which is only practical because the experiments are done in simulation). Prior work has confirmed that combining perception tasks like semantic segmentation with end-to-end driving networks yield better performance, including using a strongly related approach (Xu et al). In addition to the lack of novelty or new insights, the writing needs serious attention. For these reasons, I believe this paper is not suitable for publication at ICLR.