MITSUBISHI ELECTRIC RESEARCH LABORATORIES
https://www.merl.com

Scene-aware Interaction
Hori, Chiori; Tsuchiya, Masato; Chen, Siheng; Cherian, Anoop; Hori, Takaaki; Harsham, Bret A.;
Marks, Tim K.; Le Roux, Jonathan; Sullivan, Alan; Vetro, Anthony
TR2021-042

May 04, 2021

Abstract
The recent artificial intelligence (AI) boom and intelligent use of data acquired from various
sensors has accelerated the development of technologies needed to realize advanced humanlike capabilities in machines. AI technologies have come a long way in accurately perceiving
visual scenes and understanding speech. However, one important piece of technology is still
missing: natural and context-aware human-machine interaction, where machines understandtheir surrounding scene from the human perspective and are able to share their understanding
with humans using natural language. To bridge this communication gap, we have developed
and built a new AI system, called scene-aware interaction, that enables machines to translate
their perception and understanding of a scene and respond to it using natural language tointeract more effectively with humans. This paper introduces an example application ofscene-aware interaction to car navigation systems, which will provide drivers with intuitive
route guidance.
Society of Automotive Engineers of Japanc 2021 MERL. This work may not be copied or reproduced in whole or in part for any commercial purpose. Permission
to copy in whole or in part without payment of fee is granted for nonprofit educational and research purposes provided
that all such whole or partial copies include the following: a notice that such copying is by permission of Mitsubishi
Electric Research Laboratories, Inc.; an acknowledgment of the authors and individual contributions to the work; andall applicable portions of the copyright notice. Copying, reproduction, or republishing for any other purpose shallrequire a license with payment of fee to Mitsubishi Electric Research Laboratories, Inc. All rights reserved.
Mitsubishi Electric Research Laboratories, Inc.
201 Broadway, Cambridge, Massachusetts 02139

࣐ࣝࢳ࣮ࣔࢲࣝࢭࣥࢩࣥࢢ᝟ሗ࡟ᇶ࡙ࡃ Scene-aware Interaction ᢏ⾡ᇼᬛ⧊ ࣭ᅵᒇᨻே ࣭Siheng Chen࣭Anoop Cherian࣭ᇼ ㈗࣭᫂Bret Harsham
Tim K. Marks࣭Jonathan Le Roux࣭Alan Sullivan࣭Anthony Vetro୕⳻㟁ᶵ᝟ሗᢏ⾡⥲ྜ◊✲ᡤࠊ0LWVXELVKL(OHFWULF5HVHDUFK/DERUDWRULHV㸯ࡣࡌࡵ࡟⣙ 100 ᖺ๓࡟ࠕࣟ࣎ࢵࢺࠖ࡜࠸࠺ே㛫ࢆᶍࡋࡓᶵᲔࡀே㛫ࡢ௦ࢃࡾ࡟ປാࡍࡿ࡜࠸࠺≀ㄒࡀ๰సࡉࢀ࡚௨᮶ࠊே㛫࡜఍ヰࡋࠊே㛫ࡢࡼ࠺࡟⮬ᚊ⾜ື࡛ࡁࡿࣟ࣎ࢵࢺࡀከࡃࡢᫎ⏬ࡸ࢔ࢽ࣓࡞࡝࡟Ⓩሙࡍࡿࡼ࠺࡟࡞ࡗࡓࠋࣟ࣎ࢵࢺࡢᙧ≧ࡣேᆺ࡟␃ࡲࡽࡎ⮬ື㌴ᆺ࡞࡝ከᵝ࡛ࠊࣟ࣎ࢵࢺࡣಶᛶࢆᣢࡘே㛫࡜ࡋ࡚ᨃே໬ࡉࢀ࡚ᥥ࠿ࢀ࡚࠸ࡿࠋ⌧ᐇࡢୡ⏺࡛ࡣ AI ᢏ⾡ࡢ㠉᪂ⓗ࡞㐍Ṍ࡜ᵝࠎ࡞ࢭࣥࢧ࠿ࡽྲྀᚓࡉࢀࡓࢹ࣮ࢱࡢ▱ⓗ฼⏝࡟ࡼࡾࠊࣟ࣎ࢵࢺࡢᐇ⏝໬࡟ྥࡅࡓせ⣲ᢏ⾡ࡢ㛤Ⓨࡀᛴ㏿࡟ຍ㏿ࡋ࡚࠸ࡿࠋ≀యㄆ㆑࣭ື⏬ㄝ࣭᫂⮬↛ゝㄒ⏕ᡂ࣭㡢ኌᑐヰᢏ⾡ࡢᛶ⬟ࡣ῝ᒙᏛ⩦ࢆ⏝࠸ࡿࡇ࡜࡛๻ⓗ࡟ᨵၿࡋࠊࢱࢫࢡ࡟ࡼࡗ࡚ࡣࡍ࡛࡟ே㛫ࢆ㉸࠼ࡿ⬟ຊࢆ♧ࡋ࡚࠸ࡿࠋ౛࠼ࡤࠊከゝㄒ」ᩘヰ⪅࡟ࡼࡿྠ᫬Ⓨヰࡢ㡢ኌㄆ㆑ᢏ⾡>@࡞࡝ࡀࡑࡢⰋ࠸౛࡛࠶ࡿࠋࡇࡢࡼ࠺࡞ᢏ⾡㠉᪂ࡢ⤖ᯝࠊࢥࣥࣆ࣮ࣗࢱࡣே㛫ࡢゝⴥࢆ⌮ゎࡍࡿࡼ࠺࡟࡞ࡾࠊே㛫ࡢᣦ♧࡟ᚑ࠸࢔ࣉࣜࢣ࣮ࢩࣙࣥࢆ᧯సࡋࠊ࢖ࣥࢱ࣮ࢿࢵࢺୖ࡟࠶ࡿ⏬ീࡸ㡢ኌࡶ࣮࣮࢟࣡ࢻ࡛▐᫬࡟᳨⣴࡛ࡁࡿࡼ࠺࡟࡞ࡗࡓࠋࡋ࠿ࡋࠊᐇୡ⏺࡛ࡣ SF ᫎ⏬ࡢࣟ࣎ࢵࢺࡢࡼ࠺࡟ே㛫࡜ྠ➼ࡢ⬟ຊࢆᣢࡕே㛫࡜᥮ࢃࡿࡇ࡜ࡀ࡛ࡁࡿᶵᲔࡣᮍࡔⓎ᫂ࡉࢀ࡚࠸
࡞࠸ࠋே㛫ࡣᐇୡ⏺ࡢືⓗ࡟ኚ໬ࡍࡿ」㞧࡞஦㇟ࢆឤぬჾᐁ࠿ࡽᚓࡽࢀࡓ᝟ሗࢆ⏝࠸࡚⌮ゎࡋࠊࡉࡽ࡟ࡑࡢ⌮ゎࢆ⮬↛࡞ゝⴥ࡛⾲⌧ࡋ࡚௚ࡢே㛫࡜ពᛮ␯㏻ࡍࡿࡇ࡜ࡀ࡛ࡁࡿࡀࠊ
⌧ᅾࡢᶵᲔ࡟ࡣࡑࡢ㔜せ࡛Ḟ࠿ࡍࡇ࡜ࡢ࡛ࡁ࡞࠸ᶵ⬟ࡀ↓࠸ࠋᶵᲔࡀᐇୡ⏺ࡢ஦㇟ࢆㄆ㆑࣭⌮ゎࡋ⮬↛ゝㄒ࡛ពᛮ␯㏻ࢆࡍࡿ࡟ࡣࠊᩘ್࡛⾲⌧ࡉࢀࡓ」ᩘࡢࢭࣥࢩࣥࢢ᝟ሗࢆ⤫ྜࡋ୍࡚ࡘࡢ஦㇟࡜ࡋ࡚⌮ゎࡋࠊࡑࢀࡽࡀ⤌ࡳྜࢃࡉࡗࡓ」㞧࡞஦㇟ࢆே㛫ࡢ≧ἣ⌮ゎ࡟ᑐᛂࡋࡓ⮬↛ゝㄒ࡛⾲⌧ࡋࠊࡉࡽ࡟ே㛫࡜ࡢᑐヰࡢᩥ⬦࡜ㄆ㆑ࡋࡓ࿘ᅖࡢ≧ἣࢆྠ᫬࡟⪃៖ࡋ࡚㐺ษ࡞ᛂ⟅ᩥࢆ⏕ᡂࡍࡿᚲせࡀ࠶ࡿࠋ᪤Ꮡࡢᢏ⾡࡛ࡣࠊಶูࡢᑠつᶍ࡞஦㇟ࡢㄆ㆑࡟㝈ᐃⓗ࡞ゝㄒ⾲⌧ࢆࢡࣛࢫྡ࡜ࡋ࡚ᑐᛂࡉࡏࡿᶵ⬟࡟␃ࡲࡾࠊ⮬↛ゝㄒ⏕ᡂࢆ⏝࠸ࡓヲ⣽࡞≧ἣ⌮ゎ࡟ࡣ⛬㐲࠿ࡗࡓࠋ࡜ࡇࢁࡀࠊ㏆ᖺࡢ῝ᒙᏛ⩦ࡢ㐍Ṍ࡟ࡼࡾࢽ࣮ࣗࣛࣝࢿࢵࢺ࣮࣡ࢡ࡛ࣔࢹࣝ໬ࡉࢀࡓ㆑ูჾ㸦ࢡࣛࢫศ㢮㸧ࡸ༢ㄒࢆ୪࡭࡚ᩥࢆ⏕ᡂࡍࡿࡼ࠺࡞グྕิ⏕ᡂ⿦⨨ࠊ࠶ࡿ࠸ࡣࡑࢀࡽࢆ⤫ྜࡋࡓࣔࢹࣝࡀ༢୍ࡢࢿࢵࢺ࣮࣡ࢡ࡛⾲⌧࡛ࡁࡿࡼ࠺࡟࡞
ࡾࠊEnd-to-End Ꮫ⩦ࢆ⏝࠸࡚ධฟຊࡢࢧࣥࣉࣝࡔࡅ࡛Ꮫ⩦࡛ࡁࡿࡼ࠺࡟࡞ࡗࡓࠋࡇࡢᢏ⾡ⓗ㠉᪂࡟ࡼࡾࠊࣟ࣎ࢵࢺࡀᚲせ࡜ࡍࡿᐇୡ⏺ࡢ≧ἣ⌮ゎ࡟ᇶ࡙ࡃ㡢ኌᑐヰᶵ⬟ࡀᐇ⌧࡛ࡁࡿ㐨➽ࡀぢ࠼ࠊᡃࠎࡣ End-to-End ῝ᒙᏛ⩦ࢆ⏝࠸࡚ࠊ」ᩘࡢࢭࣥࢧࡀ཰㞟ࡋࡓ᝟ሗ㸦࣐ࣝࢳ࣮ࣔࢲࣝࢭࣥࢩࣥࢢ᝟ሗ㸧࠿ࡽ࿘ᅖࡢ≧ἣࢆᶵᲔࡀ⮬↛࡞ゝㄒ࡛⌮ゎࡋࠊே࡜ࡼࡾ෇⁥࡞ពᛮ␯㏻ ࢆ ᐇ ⌧ ࡍ ࡿ ࠕ Scene-Aware Interaction ᢏ ⾡ ࠖ ࢆ 㛤 Ⓨ ࡍ ࡿ ࡇ ࡜ ࡀ ࡛ ࡁ ࡓ ࠋ Scene-awareinteraction ᢏ⾡ࡣࠊ࣓࡛࢝ࣛ᧜ᙳࡋࡓ⏬ീ᝟ሗࠊ࣐࢖ࢡࣟࣇ࢛࡛ࣥ㞟㡢ࡋࡓ㡢㡪᝟ሗࠊ/L'$5 ࡸ࣮ࣞࢲ࣮࡛ྲྀᚓࡋࡓ఩⨨᝟ሗ࡞࡝ࡢ࣐ࣝࢳ࣮ࣔࢲࣝࢭࣥࢩࣥࢢ᝟ሗ࠿ࡽࠊఱࡀ࡝ࡇ࡛࡝ࡢࡼ࠺࡞≧ែ࡟࠶ࡿࡢ࠿ࠊㄡࡀ࡝ࡇ࡛ఱࢆࡋ࡚࠸ࡿࡢ࠿ࠊ࡜࠸ࡗࡓ࿘ᅖࡢ≧ἣࢆᶵᲔࡀ
⮬↛ゝㄒ࡛⌮ゎࡋࠊே㛫࡜ࡢ఍ヰࡢᩥ⬦ࡶ⪃៖ࡋ࡚ᛂ⟅ᩥࢆ⏕ᡂࡍࡿᢏ⾡࡛࠶ࡿࠋᮏ Sceneaware Interaction ᢏ⾡ࡣࠊࣟ࣎ࢵࢺࡸࣔࢽࢱࣜࣥࢢࢩࢫࢸ࣒࡜࠸ࡗࡓ≧ἣ⌮ゎ࡟ᇶ࡙ࡁே㛫࡜࢖ࣥࢱࣛࢡࢩࣙࣥࢆᚲせ࡜ࡍࡿᵝࠎ࡞ࢩࢫࢸ࣒࡬ࡢᛂ⏝ࡀᮇᚅ࡛ࡁࡿ⏬ᮇⓗ࡞ᢏ⾡࡛࠶ࡿࠋᮏ✏࡛ࡣࠊࡑࡢᛂ⏝౛࡜ࡋ࡚㌴㍕ࡢᵝࠎ࡞ࢭࣥࢧ࠿ࡽྲྀᚓࡉࢀࡓ」ᩘࡢࢭࣥࢧ᝟ሗ࡟
ᇶ࡙ࡁ≧ἣࢆ⌮ゎࡋ⤒㊰᱌ෆࢆ⾜࠺ࢩࢫࢸ࣒ࢆ⤂௓ࡍࡿࠋ㸰. ᮍ᮶ᆺ࣮࢝ࢼࣅࢤ࣮ࢩࣙࣥࢩࢫࢸ࣒㏆ᖺࠊከࡃࡢ࣮࢝࢖ࣥࣇ࢛ࢸ࢖࣓ࣥࣥࢺࢩࢫࢸ࣒ࡢ᧯స࡟㡢ኌᑐヰࡀᑟධࡉࢀ࡚࠸ࡿࠋ㡢ኌᑐヰࡣ㐠㌿᧯సࢆጉࡆࡿࡇ࡜ࡢ࡞࠸㠀ᖖ࡟᭷ຠ࡞࢖ࣥࢱࣛࢡࢩࣙࣥᡭẁ࡛࠶ࡿࡀࠊ⌧ᅾࢩࢫࢸ࣒ࡀᢅ࠼ࡿ▱㆑ࡣࢥࣥࣆ࣮ࣗࢱୖ࡟ಖᏑࡉࢀࡓ᪤Ꮡࡢ▱㆑࠿ࢿࢵࢺ࡞࡝࠿ࡽ㓄ಙࡉ
ࢀࡿ஺㏻≧ἣ࡞࡝㝈ᐃⓗ࡛࠶ࡿࠋᕷ㈍ࡢ࣮࢝ࢼࣅࢤ࣮ࢩࣙࣥࢩࢫࢸ࣒ࡢ୰࡟ࡣヲ⣽໬ࡉࢀࡓᆅᅗ᝟ሗ࡟ᇶ࡙ࡁၟᴗ᪋タྡ࡛᱌ෆࡍࡿࡶࡢࡶ࠶ࡿࡀࠊᮍࡔ *36 ࡢ⮬㌴఩⨨᝟ሗ࡟ᇶ࡙ࡁࠕ㸯㸮㸮m ඛࡢ஺ᕪⅬࢆྑ࡟᭤ࡀࡗ࡚ࡃࡔࡉ࠸ࠋࠖ࡜࠸ࡗࡓ㊥㞳ࢆ⏝࠸ࡓ⤒㊰᱌ෆࡍࡿࢩࢫࢸ࣒ࡀከ࠸ࠋ୍᪉ࠊຓᡭᖍࡢே㛫ࡀ⤒㊰᱌ෆࢆࡍࡿሙྜࠊࠕ㒑౽࣏ࢫࢺࡢ࡜ࡇࢁ࡛ྑᢡࡋ࡚ࡃࡔࡉ࠸ࠋࠖࠕ஺ᕪⅬ࡛ᕥᢡࡋ࡚࠸ࡿ㖟Ⰽࡢ㌴࡟⥆࠸࡚ᕥᢡࡋ࡚ࡃࡔࡉ࠸ࠋࠖࡢࡼ࠺࡟௒ぢ࠼࡚࠸ࡿ≉ᚩⓗ࡞┠༳࡜࡞ࡿ≀యࢆ⏝࠸࡚┤ឤⓗ࡟ㄝ᫂ࡋࠊࡉࡽ࡟㐍⾜᪉ྥ࡟༴㝤ࡀ㏕ࢀࡤࠕṌ⾜⪅ࡀ㐨ࢆᶓษࢁ࠺࡜ࡋ࡚࠸ࡲࡍࠋࠖ࡜࠸ࡗࡓࡼ࠺࡟ὀពႏ㉳ࡶࡋ࡚ࡃࢀࡿࠋຓᡭᖍࡢே㛫ࡣࠊ้ࠎ࡜ኚ໬ࡍࡿ≧ἣ࡟ᛂࡌ࡚㐠㌿⪅ࡀㄆ㆑ࡋࡸࡍ࠸≀యࢆ⏝࠸࡚⤒㊰᱌ෆࢆࡍࡿࠋࡇࡢࡼ࠺࡞┤ឤⓗ࡞⤒㊰᱌ෆࡀྍ⬟࡟࡞ࡿࡢࡣࠊ㐠㌿⪅࡜ຓᡭᖍࡢே㛫ࡀྠࡌ⎔ቃୗ࡛≧ἣ࡟ᑐࡍࡿㄆ㆑࡜⌮ゎࢆඹ᭷ࡋ࡚࠸ࡿ࠿ࡽ࡛࠶ࡿࠋᶵᲔ࡛ຓᡭᖍࡢே㛫࡜ྠ➼ࡢᶵ⬟ࢆᐇ⌧ࡍࡿ࡟ࡣࠊᐇୡ⏺ࡢືⓗ࡟ኚ໬ࡍࡿ஦㇟ࢆㄆ㆑ࡋࠊே㛫ࡢどᗙ࡛ᤊ࠼ࡓ≧ἣ⌮ゎࢆ⾜࠸ࠊே㛫࡜⮬↛ゝㄒ࡛ពᛮ␯㏻ࢆࡍࡿ Scene-aware Interaction ᢏ⾡ࡀᚲせ࡜࡞ࡿࠋᅗ㸯࡟ Scene-aware Interaction ᢏ⾡ࢆ⏝࠸ࡓ⤒㊰᱌ෆࢩࢫࢸ࣒ࡢᴫせࢆ♧ࡍࠋᅗ㸯 Scene-aware Interaction ᢏ⾡ࢆ⏝࠸ࡓ⤒㊰᱌ෆࢩࢫࢸ࣒ࡢᴫせ⌧ᅾࠊ኱ከᩘࡢ⮬ື㌴ࡀᏳ඲ࡢࡓࡵࡢ㐠㌿ᨭ᥼࡜⮬ືไᚚࢆ┠ⓗ࡜ࡋ࡚ከࡃࡢࢭࣥࢧࢆᦚ㍕ࡋ࡚࠸ࡿࠋ㌴እ࡟࣓࢝ࣛࠊ࣑ࣜἼ࣮ࣞࢲࠊ㉸㡢Ἴࢭࣥࢧࡀ⿦ഛࡉࢀࠊ⮬ື㉮⾜ไᚚࠊ⮬ື⥭ᛴࣈ࣮ࣞ࢟ࠊ㌴⥺⥔ᣢཬࡧ㥔㌴⿵ຓ࡟⏝࠸ࡽࢀ࡚࠾ࡾࠊ㌴ෆ࣓࢝ࣛࡣ㐠㌿⪅ࡢ೺ᗣ≧ែࢆࣔࢽࢱࡍࡿࠋࡇࢀࡽࡢࢭࣥࢧࡣṚゅ࡟࠶ࡿ㌴ࡢᏑᅾࢆࣅ࣮ࣉ㡢࡛ࠊ㌴⥺㐓⬺ࢆࣁࣥࢻࣝࡢ᣺ື࡛㐠㌿⪅࡟▱ࡽࡏࡿࡀࠊᐇୡ⏺ࡢ≧ἣ࡟ࡘ࠸࡚⮬↛ゝㄒࢆ⏝࠸࡚ᶵᲔࡀ㐠㌿⪅࡜࢖ࣥࢱࣛࢡ
ࢩࣙࣥࢆ⾜࠺┠ⓗ࡛ࡣ⏝࠸ࡽࢀ࡚࠸࡞࠸ࠋ㐠㌿⪅࡜࢖ࣥࢱࣛࢡࢩࣙࣥࡍࡿࡢ࡟࡝ࡢࡼ࠺࡞᪉ἲࡀⰋ࠸ࡢ࠿࡜࠸࠺㆟ㄽࡀ࠶ࡿࡀࠊ᭱㏆ࡢ◊✲࡛ࡣ⮬↛ゝㄒ࡛⾲⌧ࡉࢀࡓヲ⣽࡞࣓ࢵࢭ࣮ࢪࡀࡼࡾຠᯝⓗ࡟ព࿡᝟ሗࢆఏ㐩ࡍࡿ࡜࠸࠺ࡇ࡜ࡀ♧ࡉࢀ࡚࠾ࡾࠊ⥭ᛴ஦ែࢆ㝖ࡅࡤࠊ┤ឤⓗ࡛≉ู࡞カ⦎ࢆᚲせ࡜ࡋ࡞࠸ࡇ࡜࠿ࡽ᭱ࡶᮃࡲࡋࡃࠊ୍⯡࡟ཷࡅධࢀࡽࢀ࡚࠸ࡿ᪉ἲ࡛࠶ࡿ࡜ሗ࿌ࡉࢀ࡚࠸ࡿ>@ࠋ㸱࣐ࣝࢳ࣮ࣔࢲࣝࢭࣥࢩࣥࢢ᝟ሗ࡟ᇶ࡙ࡃ≧ἣ⌮ゎᐇୡ⏺ࡢ஦㇟࡟ࡘ࠸࡚⮬↛ゝㄒ࡛࢖ࣥࢱࣛࢡࢩࣙࣥࢆ⾜࠺ࢩࢫࢸ࣒࡟࠾࠸࡚᭱ࡶ㔜せ࡞ᶵ⬟ࡣࠊ≀యࡸ஦㇟ࢆㄆ㆑ࡍࡿᶵ⬟࡜ࠊࡑࡢࢭࣥࢩࣥࢢ᝟ሗࢆே㛫ࡢ⌮ゎ࡟ᑐᛂࡋࡓ⮬↛ゝㄒ࡟⩻ヂࡍࡿᶵ⬟࡛࠶ࡿࠋᮏ❶࡛ࡣࠊ⏬ീㄆ㆑ࡸ㡢㡪ㄆ㆑࡞࡝ࡢ」ᩘࡢࢭࣥࢩࣥࢢ᝟ሗࢆ⏝࠸ࡓ≀య࣭஦㇟ㄆ㆑ᢏ⾡ࠊཬࡧࡑࢀࡽࡢㄆ㆑ෆᐜࢆ⮬↛ゝㄒ࡟ኚ᥮ࡍࡿ⣔ิኚ᥮ᢏ⾡ࢆ⤂௓ࡍࡿࠋ⏬ീ≉ᚩ㔞┤ឤⓗ࡞⤒㊰᱌ෆ࡟Ḟ࠿ࡍࡇ࡜ࡀ࡛ࡁ࡞࠸ࢭࣥࢩࣥࢢ᝟ሗ࡜ࡋ࡚࣓࢝ࣛࡢᫎീࡀ࠶ࡿࠋຓᡭᖍࡢே㛫࡟ࡼࡿ᱌ෆࢆᶍࡋࡓᮍ᮶ᆺࡢ࣮࢝ࢼࣅࢤ࣮ࢩࣙࣥࢩࢫࢸ࣒ࡣࠊ⏬ീ᝟ሗ࠿ࡽ㐍⾜᪉ྥࢆ≉ᚩ࡙ࡅࡿࣛࣥࢻ࣐࣮ࢡࢆ᥈ࡋࠊ⮬㌴࡜ࣛࣥࢻ࣐࣮ࢡ࠶ࡿ࠸ࡣࣛࣥࢻ࣐࣮ࢡ㛫ࡢ఩⨨㛵ಀࢆ⏝࠸࡚᱌ෆᩥࢆ⏕ᡂࡋࠊ㐍㊰ࢆ㐽ࡿ≀యࡀ࠶ࢀࡤὀពႏ㉳ࢆಁࡍᩥࢆ⏕ᡂࡍࡿᚲせࡀ࠶ࡿࠋࡇࢀࡽࡢࢩࢫࢸ࣒ࡢ⏕ᡂᩥ࡛⏝࠸ࡽࢀࡿ≀యࡸ஦㇟ࡢㄆ㆑ࡣࠊ≀యㄆ㆑࣭ືయ ࢺ ࣛ ࢵ ࢟ ࣥ ࢢ 㸦 2EMHFW 'HWHFWLRQ DQG 7UDFNLQJ 㸧 >@ ࠊ ព ࿡ ⓗ 㡿 ᇦ ศ ๭ 6HPDQWLF
6HJPHQWDWLRQ >@>@ࠊ῝ᗘ᥎ᐃ 'HSWK HVWLPDWLRQ ࠊ6FHQHJUDSK ⾲⌧ࢆ⏝࠸ࡓ≀య㛫ࡢ఩⨨㛵ಀ᥎ᐃ>@ࠊ ḟඖⅬ⩌ '3RLQW&ORXG ࡟ᇶ࡙ࡃືస᥎ᐃ>@࡞࡝ࡢ⏬ീฎ⌮ᢏ⾡࡟ࡼࡗ࡚ᐇ⌧ྍ⬟࡛࠶ࡿࠋᅗ㸰࡟㌴㍕࣓࢝ࣛࡢ⏬ീฎ⌮⤖ᯝࡢ౛ࢆ♧ࡍࠋਦ̐ য়‫ڱ‬ཀྵմ͹ࠞ‫͵ͳڎ‬Ζಝ௅பड़Νࢩ͓Ζժ଀ॴཀྵٗढ़
ᕥୖ㸸㌴㍕࣓࢝ࣛᫎീࡢ๓᪉᪉ྥࡢ㢼ᬒ࡟࠾ࡅࡿᵝࠎ࡞✀㢮ࡢ≀యㄆ㆑࡜㏣㊧⤖ᯝࠊ ྑୖ㸸⮬ື㌴ࠊ㐨㊰ࠊṌ⾜⪅࡜࠸ࡗࡓࣆࢡࢭࣝ༢఩࡛ࡢព࿡ⓗ㡿ᇦศ๭⤖ᯝࠊᕥୗ㸸⮬ື㌴࡜௚ࡢ≀యࡢ㊥㞳ࡢ㛵ಀࢆ♧ࡍ῝ᗘ᥎ᐃ⤖ᯝ; ྑୗ㸸6FHQHJUDSK ⾲⌧࡟ᇶ࡙ࡃ≀య㛫ࡢ఩⨨㛵ಀࡢ᥎ᐃ⤖ᯝᮏࢩࢫࢸ࣒ࡣࠊ㐠㌿⪅ࡀ┤ឤⓗ࡟ㄆ㆑ࡍࡿࡇ࡜ࡀ࡛ࡁࡿࣛࣥࢻ࣐࣮ࢡࢆ⏝࠸࡚⤒㊰᱌ෆࢆ⾜࠺ࡇ࡜࠿ࡽࠊࢩࢫࢸ࣒ࡣ≀యㄆ㆑ࡉࢀࡓ⭾኱࡞ᩘࡢ≀యࡢ୰࠿ࡽ᭱ࡶ┤ឤⓗ࡟ㄆ㆑࡛ࡁࡿ≀యࢆ㔜せ࡞≀య࡜ࡋ࡚㑅ᢥࡍࡿᚲせࡀ࠶ࡿࠋࣛࣥࢻ࣐࣮ࢡ࡜ࡋ࡚ࡢ㔜せᗘࡣࠊ≀యㄆ㆑ࡢ⤖ᯝ࡟ᇶ࡙ࡁ≀యࡢ✀㢮ࠊᒓᛶࠊࢧ࢖ࢬࠊ఩⨨ࠊ஺ᕪⅬ࠿ࡽࡢ㊥㞳ࠊ࠾ࡼࡧ㐍⾜᪉ྥ࡞࡝ࡢ≉ᚩ࡟ᇶ࡙ࡁࢽ࣮ࣗࣛࣝᅇᖐࣔࢹࣝࢆ⏝࠸࡚Ỵᐃࡉࢀࡿࠋ㊥㞳ࡀ㐲ࡅࢀࡤ኱ࡁࡃ┠ࢆᘬࡃ≀యࠊᕥᢡࡢ㝿ࡣ஺ᕪⅬࡢᕥഃ࡟࠶ࡿ௚࡜ࡣ≉ᚩࡀ␗࡞ࡿ≀యࠊ༑ศ࡟≉ᚩⓗ࡞ࣛࣥࢻ࣐࣮ࢡࡀ↓࠸ሙྜ࡟ࡣᚑ᮶㏻ࡾ஺ᕪⅬ࠿ࡽࡢ㊥㞳ࢆ⏝࠸ࡿࠋᗑࡢ┳ᯈࡸ⮬ື㌴ࡢࣟࢦ
࡞࡝ࢆㄆ㆑ࡍࡿࡇ࡜ࡼࡾࠊࡉࡽ࡟ヲ⣽࡞⤒㊰᱌ෆࡀྍ⬟࡜࡞ࡿࠋࡉࡽ࡟ືయࢺࣛࢵ࢟ࣥࢢࡢ≉ᚩ࡜⤌ࡳྜࢃࡏࡿࡇ࡜࡛ࠊ㐍⾜᪉ྥ࡜ྠ᪉ྥ࡟㐍ࡴ๓᪉᪉ྥࡢ㌴ࢆ≉ᐃࡋ࡚┠ᶆ≀࡜ࡋࠊ㐍㊰࡟㐍ධࡍࡿ≀యࢆண ࡋ࡚ඃඛⓗ࡟ὀពႏ㉳ࢆ⾜࠺ࠋᅗ㸱࡟ືయࢺࣛࢵ࢟ࣥࢢࢆ⏝࠸ࡓ⤒㊰᱌ෆࡢ౛ࢆ♧ࡍࠋᅗ㸱ືయࢺࣛࢵ࢟ࣥࢢࢆ⏝࠸ࡓ⤒㊰᱌ෆ࡜ὀពႏ㉳ࣛࣥࢻ࣐࣮ࢡࡢ」ᩘࡢೃ⿵ࡢ୰࠿ࡽ᭱ࡶ≉ᚩⓗ࡞≀యࢆ㑅ࡪ㝿ࠊ᭱ࡶ⪃៖ࡍ࡭ࡁ᝟ሗࡣ≀య㛫ࡢ㊥㞳࡟ᇶ࡙ࡃ఩⨨㛵ಀ࡛࠶ࡿࠋ⌧ᅾࡢ *36 ࡸ༢║࣓࢝ࣛࡢࡳࢆ⏝࠸ࡓ≀యࡢ఩⨨㛵ಀ᥎ᐃ࡟ࡣ㝈⏺ࡀ࠶ࡾࠊᑗ᮶ⓗ࡟ *166 ཷಙᶵࡸ &9; ࡜࠸ࡗࡓ㐨㊰࢖ࣥࣇࣛ࡜ࡢ㏻ಙ࡟ࡼࡾ
㧗⢭ᗘ࡞⮬㌴఩⨨᝟ሗࡢྲྀᚓࡀᮇᚅ࡛ࡁࡿࡀࠊ⌧᫬Ⅼ࡛฼⏝ࡍࡿࡇ࡜ࡣ࡛ࡁ࡞࠸ࠋ௒ᅇࡢᐇ㦂࡛ࡣࠊ⮬ື㐠㌿ᢏ⾡ࡢࡓࡵࡢ ' ࣐ࢵࣆࣥࢢᢏ⾡ࢆ⏝࠸࡚ࠊ࣓࢝ࣛ⏬ീ࡜ /L'$5 ⏬ീ࠿ࡽ ' ࣐ࢵࣉୖ࡛≀యࡢ఩⨨ࢆ᥎ᐃࡍࡿ᪉ἲࢆ㐺⏝ࡍࡿࡇ࡜࡜ࡋࡓ>@ࠋᡃࠎࡣ LiDARࠊ࣓࢝ࣛࠊIMUs ࢆᦚ㍕ࡋࡓ ' ࣐ࢵࣉసᡂࡢࡓࡵࡢ Mobile Mapping System (MMS)ࢆ⏝࠸࡚ࢹ࣮ࢱ཰㞟ࢆ⾜࡞ࡗࡓࠋᅗ㸱࡟ 006 ࡛ྲྀᚓࡋࡓࢹ࣮ࢱ࡟ᇶ࡙ࡃ≀యࡢ఩⨨᥎ᐃࡢ౛ࢆ♧ࡍࠋ࣓࢝ࣛ⏬
ീ࡛ㄆ㆑ࡉࢀࡓ≀య࡜ /L'$5 ࡛ྲྀᚓࡋࡓ ' ࣏࢖ࣥࢺࢡࣛ࢘ࢻୖࡢྠ୍ࡢ≀యࢆᑐᛂ࡙ࡅࡿࡇ࡜࡟ࡼࡾࠊ' ࣏࢖ࣥࢺࢡࣛ࢘ࢻ࠿ࡽ෌ᵓ⠏ࡉࢀࡓࢺࢵࣉࢲ࣐࢘ࣥࢵࣉୖ࡛≀యࡢṇ☜࡞఩⨨ࢆྲྀᚓࡍࡿࡇ࡜ࡀྍ⬟࡜࡞ࡿࠋ

3D Point CloudCamera View

Top-Down Mapᅗ㸱006 ࢆ⏝࠸࡚ྲྀᚓࡉࢀࡓ㸱' ࣏࢖ࣥࢺࢡࣛ࢘ࢻ࡜࣓࢝ࣛ⏬ീ࠾ࡼࡧࢺࢵࣉࢲ࣐࢘ࣥࢵࣉ࣐ࣝࢳ࣮ࣔࢲࣝࢭࣥࢭࢩࣥࢢ᝟ሗே㛫࡜ྠ➼௨ୖ࡟࿘ᅖࡢ஦㇟ࢆㄆ㆑ࡍࡿࡓࡵ࡟ࡣࠊ⏬ീ᝟ሗ࡟␃ࡲࡽࡎࠊ㡢㡪᝟ሗ࡞࡝ᵝࠎ࡞ࢭࣥࢩࣥࢢ᝟ሗࡶ⪃៖ࡍࡿᚲせࡀ࠶ࡿࠋ౛࠼ࡤࠊ⏬ീ࡛ᩆᛴ㌴ࡣㄆ㆑࡛ࡁ࡚࠸࡞࠸
ࡀ㡢㡪᝟ሗ࠿ࡽࡑࡢືྥࡀ᥎ᐃ࡛ࡁࠊࠕᚋ᪉࠿ࡽᩆᛴ㌴ࡀ㏆࡙࠸࡚᮶ࡲࡍࠋࠖ࡜࠸ࡗࡓㄝ᫂ᩥࡀ⏕ᡂ࡛ࡁࡿࠋࡇࡢሙྜࠕᚋ᪉ࠖࠕᩆᛴ㌴ࠖ࡜࠸࠺༢ㄒࡣ⏬ീࡢ≉ᚩ㔞ࡢࡳ࡛⏕ᡂࡍࡿࡇ࡜ࡣ࡛ࡁ࡞࠸ࠋ࣐ࣝࢳ࣮ࣔࢲࣝࢭࣥࢩࣥࢢ᝟ሗࢆ⏝࠸ࡿࡇ࡜࡛ே㛫ྠ➼࠶ࡿ࠸ࡣࡑࢀ
௨ୖࡢ≧ἣ⌮ゎࡀྍ⬟࡟࡞ࡿࠋ」ᩘࡢಙྕࢆࡲ࡜ࡵୖࡆ୍࡚ࡘࡢ஦㇟࡜ࡋ࡚⌮ゎࡍࡿ࡟ࡣࠊ࣐ࣝࢳ࣮ࣔࢲࣝࢭࣥࢩࣥࢢ᝟ሗࢆ⤫ྜࡋ࡚᪂ࡓ࡞ព࿡✵㛫ࢆᵓ⠏ࡋࠊࡑࡢ✵㛫ୖ࡛༢ㄒิࢆ≉ᚩ࡙ࡅࡿᚲせࡀ࠶ࡿࠋᡃࠎࡣ༢ㄒẖ࡟≉ᚩⓗ࡞ࢭࣥࢩࣥࢢ᝟ሗ࡟␗࡞ࡿ㔜ࡳࡀ࠿࠿ࡿ
ࡼ࠺࡟⮬ືᏛ⩦ࡍࡿ࣐ࣝࢳ࣮ࣔࢲ࣭ࣝ࢔ࢸࣥࢩࣙࣥἲࢆ⪃᱌ࡋࡓ>@ࠋࡑࡢ⤖ᯝࠊ␗࡞ࡿࢭࣥࢩࣥࢢ᝟ሗࡀ┦஫⿵᏶ࡍࡿࡇ࡜࡛ࠊࡼࡾṇࡋࡃ≧ἣ⌮ゎࡍࡿࡇ࡜ࡀྍ⬟࡜࡞ࡿࠋ
⣔ิኚ᥮ࣔࢹࣝ」ᩘࡢࢭࣥࢩࣥࢢ᝟ሗ࠿ࡽ༢ㄒิࢆ⏕ᡂࡍࡿ᪉ἲ࡜ࡋ࡚ࠊ⣔ิ࠿ࡽ⣔ิ࡟ኚ᥮ࡍࡿ⣔ิኚ᥮ࣔࢹࣝ㸦6HTXHQFHWRVHTXHQFHPRGHO㸧ࡀᥦ᱌ࡉࢀࡓࠋཎゝㄒࡢ༢ㄒิࢆධຊࡋ┠ⓗゝㄒࡢ༢ㄒิࢆฟຊࡍࡿᶵᲔ⩻ヂ>@ࡸ㡢ኌಙྕ⣔ิࢆධຊ࡜ࡋ࡚᭩ࡁ㉳ࡇࡋࡢ༢ㄒิࢆฟຊࡍࡿ㡢ኌㄆ㆑>@࡞࡝࡟┒ࢇ࡟⏝࠸ࡽࢀࠊື⏬࡟࠾ࡅࡿ஦㇟ࢆ⮬↛ゝㄒ࡛ㄝ᫂ࡍࡿື⏬ㄝ᫂
㸦9LGHR 'HVFULSWLRQ㸧ᢏ⾡࡟ࡶ㐺⏝ࡉࢀࡓ>@ࠋࡇࢀࡣࠊື⏬ࡢ≉ᚩ㔞ࢆධຊ࡜ࡋ࡚ㄝ᫂ᩥࡢ༢ㄒิࢆฟຊࡍࡿࡇ࡜࡛ᐇ⌧ࡋࡓࡶࡢ࡛࠶ࡿࠋᡃࠎࡣ⏬ീ᝟ሗࡔࡅ࡛࡞ࡃ㡢㡪᝟ሗࡶ࣐ࣝࢳ࣮ࣔࢲ࣭ࣝ࢔ࢸࣥࢩࣙࣥἲࢆ⏝࠸࡚⪃៖ࡍࡿࡇ࡜࡟ࡼࡾࠊ<RX7XEH ື⏬ㄝ᫂ࡢඹ㏻ࢱ
ࢫࢡ࡛ᛶ⬟ࡀᨵၿࡉࢀࡿࡇ࡜ࢆ♧ࡋࡓ>@ࠋ㸲㸬ࢽ࣮ࣗࣛࣝᑐヰࣔࢹࣝ࡟ᇶ࡙ࡃ 6FHQHDZDUH,QWHUDFWLRQ㡢ኌᑐヰࢩࢫࢸ࣒ࡢ◊✲ࡣேᡭ࡟ࡼࡿ࣮ࣘࢨពᅗࡢࣛ࣋ࣝ௜ࡅࡀᚲ㡲ࡢࡓࡵࢩࢫࢸ࣒ࢆ኱つᶍ໬ࡍࡿࡇ࡜ࡀ㠀ᖖ࡟㞴ࡋ࠸࡜࠸࠺ၥ㢟ࡀ࠶ࡗࡓࠋ㏆ᖺࡢ⣔ิኚ᥮ࡢ῝ᒙᏛ⩦ࣔࢹࣝ࡟ࡼࡾࠊ
ᑐヰࡢᒚṔ࡜࣮ࣘࢨࡢධຊ࡟ᇶ࡙ࡁࠊࢩࢫࢸ࣒ࡢᛂ⟅ࢆ⮬ື⏕ᡂࡍࡿᢏ⾡ࡀ᳨ウࡉࢀ࡚࠸ࡿࠋᡃࠎࡣ  ᖺࡼࡾ 'LDORJ6\VWHP7HFKQRORJ\&KDOOHQJH '67& ࢆ୺ദࡋࠊ࢝ࢫࢱ࣐࣮ࢧ࣮ࣅࢫࡢࢸ࢟ࢫࢺࢳࣕࢵࢺ࡞࡝ࡢᑐヰࢹ࣮ࢱࢆ⏝࠸࡚ (QGWR(QG ῝ᒙᏛ⩦࡟ࡼࡿᛂ⟅⮬ື⏕ᡂࡢ◊✲㛤Ⓨࢆ⾜࡞ࡗ࡚࠸ࡿ>@ࠋࡋ࠿ࡋ࡞ࡀࡽࠊࡇࢀࡽࡢᑐヰࢩࢫࢸ࣒ࡣ඲࡚ࡢ᝟ሗࡀࢸ࢟ࢫࢺ໬ࡉࢀࡓᏛ⩦ࢹ࣮ࢱ࡛࠶ࡿࡇ࡜ࢆ๓ᥦ࡜ࡋ࡚࠾ࡾࠊ㏲ḟⓗ࡟ኚ໬ࡍࡿ࿘ᅖࡢ≧ἣࢆࢭࣥࢩࣥࢢࡋࡓ᝟ሗ࡟ᇶ࡙ࡃᑐヰࢆᑐ㇟࡜ࡋ࡚࠸࡞࠸ࠋᡃࠎࡣࠊࢽ࣮ࣗࣛࣝࢿࢵࢺ࣮࣡ࢡ࡟ᇶ࡙ࡃື⏬ㄝ᫂ᢏ⾡࡜ࡇࡢᑐヰᢏ⾡ࢆ୍య໬ࡋ࡚ (QGWR(QG ࡛Ꮫ⩦ࡉࡏࡿ 6FHQH$ZDUH,QWHUDFWLRQ ᢏ⾡ࢆ⪃᱌ࡋࠊ$XGLR9LVXDO6FHQHDZDUH'LDORJ $96' ࡜࠸࠺◊✲ㄢ㢟࡟ྲྀࡾ⤌ࢇ࡛࠸ࡿࠋᅗ㸲࡟ $96' ࡢࡓࡵࡢࢽ࣮ࣗࣛࣝࢿࢵࢺ࣮࣡ࢡࡢᵓᡂࢆ♧ࡍࠋᮏ◊✲ㄢ㢟ࢆⓎᒎࡉࡏࡿࡓࡵࠊ'67& ୺ദࡢ࣮࣡ࢡࢩࣙࢵࣉ࡟࡚ $96' ࢆࢱࢫࢡ࡜ࡋࡓ➇தⓗ◊✲㛤Ⓨࢆ⾜࡞࠸>@>@ࠊᡃࠎࡣᅗ㸳࡟♧ࡍ '\QDPLF5HSUHVHQWDWLRQ/HDUQLQJ ࢆᥦ᱌ࡋ࡚࠸ࡿ>@ࠋᅗ㸲$96' ࡢࡓࡵࡢࢽ࣮ࣗࣛࣝࢿࢵࢺ࣮࣡ࢡࡢᵓᡂᅗ㸳$96' ࡢࡓࡵࡢ '\QDPLF5HSUHVHQWDWLRQ/HDUQLQJ௒ᚋࡢᒎ㛤ᮏ✏࡛ࡣ 6FHQHDZDUH,QWHUDFWLRQ ᢏ⾡ࢆ⏝࠸ࡓḟୡ௦ࡢ࣮࢝ࢼࣅࢤ࣮ࢩࣙࣥࢩࢫࢸ࣒ࢆ⤂௓ࡋࡓ https://www.youtube.com/watch?v=zcA6p4DEIHU ࠋ6FHQHDZDUH ,QWHUDFWLRQ ᢏ⾡ࡣࠊ⮬ື㐠㌿࡬ࡢᛂ⏝ࠊ↓ே᪋タࡢ┘どࠊබඹ✵㛫࡟࠾ࡅࡿぢᏲࡾࠊ⊂ᒃࢩࢽ࢔ࡢ࣮ࣜࣔࢺࢧ࣏࣮ࢺࡸࠊࣟ࣎ࢵࢺࡀே㛫࡜ඹྠ࡛సᴗࢆࡍࡿሙ㠃࡞࡝ࠊᵝࠎ࡞฼⏝ࡀᮇᚅࡉࢀ࡚࠸ࡿࠋཧ⪃ᩥ⊩
[1] Seki, H., Hori, T., Watanabe, S., Le Roux, J., Hershey, J.R., “End-to-end multilingual multi-speakerspeech recognition,” in Proc. ISCA Interspeech, Sep. 2019, pp. 3755–3759.
[2] Bazilinskyy, P., Petermeijer, S.M., Petrovych, V., Dodou, D. and de Winter, J.C.,“Take-over requests inhighly automated driving: A crowdsourcing survey on auditory, vibrotactile, and visual displays,”
Transportation Research Part F: Traffic Psychology and Behaviour, Volume 56, 2018, pp. 82–98.
[3] Broad, A., Jones, M., & Lee, T. Y., “Recurrent Multi-frame Single Shot Detector for Video Object Detection,”
In BMVC (p. 94), 2018, September.

[4] Yu, X., Chaturvedi, S., Feng, C., Taguchi, Y., Lee, T.-Y., Fernandes, C., Ramalingam, S., "VLASE:Vehicle Localization by Aggregating Semantic Edges," in Proc. IEEE/RSJ International Conference onIntelligent Robots and Systems (IROS), Oct. 2018, pp. 3196-3203.
[5] Yu, Z., Feng, C., Liu, M.-Y., Ramalingam, S., "CASENet: Deep Category-Aware Semantic Edge
Detection," in Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Jul. 2017.
[6] Geng, S., Gao, P., Chatterjee, M., Hori, C., Le Roux, J., Zhang, Y., Li, H., Cherian, A., "Dynamic Graph
Representation Learning for Video Dialog via Multi-Modal Shuffled Transformers," in Proc. AAAI
Conference on Artificial Intelligence, Feb. 2021.
[7] Wu, P., Chen, S., "MotionNet: Joint Perception and Motion Prediction for Autonomous Driving
Based on Bird’s Eye View Maps," in Proc. IEEE Conference on Computer Vision and Pattern
Recognition (CVPR), Jun. 2020.
[8] Chen, S., Liu, B., Feng, C., Vallespi-Gonzalez, C., Wellington, C., "3D Point Cloud Processing andLearning for Autonomous Driving," IEEE Signal Processing Magazine, May 2020.
[9] Bahdanau, D., Cho, K., and Bengio, Y., "Neural machine translation by jointly learning to align andtranslate," in Proc. International Conference on Learning Representations (ICLR), May 2015.
[10] Chorowski, J., Bahdanau, D., Cho, K., and Bengio, Y., "End-to-end continuous speech recognition
using attention-based recurrent NN: First results," in Proc. NIPS 2014 Workshop on Deep Learning,
Dec. 2014.
[11] Mao J, Xu W, Yang Y, Wang J, Huang Z, Yuille A., “Deep captioning with multimodal recurrent
neural networks (m-RNN),” in Proc. International Conference on Learning Representations (ICLR), May
2015.
[12] Hori, C., Hori, T., Lee, T., Zhang, Z., Harsham, B., Hershey, J.R., Marks, T.K., Sumi, K., “AttentionBased Multimodal Fusion for Video Description,” Proceedings of the IEEE International Conference on
Computer Vision (ICCV), Oct. 2017, pp. 4193-4202.

[13] Hori, C., Perez, J., Yoshino, K., Kim, S. (Eds.), “Results from the 6th Dialog System Technology
Challenge (DSTC6)” (Special Issue), Computer Speech and Language, 2019.
[14] Hori, C., Yoshino, K., D’Haro, L.F., Galley, M., Polymenakos, L.C. (Eds.), “The 7th Dialog System
Technology Challenge (DSTC7)” (Special Issue), Computer Speech and Language, 2020.
[15] Kim, S., Schulz, H., Gunasekara, C., Hori, C., Rastogi, A., D’Haro, L.F. (Eds.), “The 8th Dialog System
Technology Challenge (DSTC8)” (Special Issue), IEEE/ACM Transactions on Audio Speech and Language
Processing, to appear in 2021.