別紙41 (課程博士(英文))Date of submission: July 3'd,2023
rof. Jun Miura

Depa巾nentof

Computer science

StudentlD Nulnberrof. shigeru

and Engineering
Pplicant's nalne

209302SupeNi50r

Urlyalna

Oskar Natan

Abstract (Doctor)
Title ofThesis

Deep perception・Action coupling a11d sensorFusion for End・to・endAutonomous Driving

Approx.80o words
In autonomous ddving, how should weintegrateperception and action components properly? what
do we need to process and fuse multi・modal data from various sensors? 1n this study, we aim toanswer these questions by conducting some experiments on end・to・end autonomous ddving toachieve exce11ent ddvability in complex environments under diverse conditions and scenados.
End、to、end autonomous ddving is a method that a110ws an autopilot modelto directly use raw
Sensor data as the inputs and outputs the low‑1evel controlsuch as steedng a11gle and throttle level.
Since manualintegration forjoining perception and action parts is n010nger needed, this method
has become a preferred approach as the model C釘l exalnine the necessity of infonnation a11 byitself. Moreover, this method can be combined with imitation leaming or behavior cloning which

Can be done easily with the supervised leaming technique. By usin8 the end・to‑end behavior
Cloning strategy, we can create a single deep learning modelto mimic an expert drivermanipulation
On vehide controHn handHng compHcated situations, which can be simulated using a simulator ordedved 丘om pubHcly available datasets to endch the driving expedences. Although this method
has plenty of benefits in pedorming huma11‑1ike autonomous ddving, there are two fundamental
issues that remain and need to be addressed carefU11y. First, since the model has multiple outputs

as it deals with perception and controltasks simultaneously atthe salne time, we must ensure thatthe solution for each task ca11 be leamed at an equal pace. This is necessary to preventthe modelfrom tending to focus only on one specific task dudng the training process. second, we need todesign a complex network architecture using the conect layerS 飢d tune some hyperparameters
accordingly, so that the model can handle multゆle input data with different modalities. ThisCha11enge arises as we must create a compact unified modelto avoid the burden of linking someindependent task・speciac modules, which may lead to infonnation loss dudng the process offorming a connected modularsystem.

To tackle those issues, we propose an end・to‑end multi・task modelthat ca11 extract meaningful
information from a set of observations retdeved by vehicle sensors and solve multiple tasks fromthe perception stage to the action stage in one forward pass. W'e also propose an adaptive loss
Weighting algodthm to ba1飢Ce the learning signal for each task equa11y. Then, we evaluate ourProposed model by doing ablation 鬮d comparative studies with other models for clearer
Perfonnance justiflcation. The expedmental results show that our model achieves supedor
Performance in many cdteda and aspects of ddving. To better clarify the findings, this study isConducted gradua11y and step‑by・step from the perception parts to the integration ofperception andaction parts, from ptedictin今 Some driving records to pedorming automated ddving, a11d from theUtilization ofsimulation programsto real‑world implementation on arobotic vehicle. Furthermore,as part of this thesis a11d to support future studies, we share the codes and data used in ourPublications publicly at https:ガgithub.convoskamatan. see the publication page for more details、