機械知覚&ロボティクスグループ
中部大学 工学研究科 ロボット理工学専攻

2020年度 藤吉研究室 卒業論文発表 アブストラクト
Deep Learning Fine-Grained Object Classification

視線情報を用いたアンサンブル推論による詳細画像識別の高精度化
谷中 亮佑
Deep Learning Object Detection Compositional models

2段階判定によるオクルージョンに対応した歩行者検出秋山 賢樹
Deep Learning Attention Branch Network Data Augmentation

Mixup拡張に対するABNによるアテンションの獲得と認識精度への影響
家崎 晃
Deep Learning Attention Branch Network Attention Mining

Attention mining branchを導入したABN によるアテンションマップの最適化
岩吉 孝明
Deep Learning Attention Branch Network Adversarial Examples

Attention branchの匿名化によるAdversarial Examplesに頑健なABN山川 翔太郎Deep Learning Knowledge Distillation

知識蒸留におけるMultiple eacher Networkの選択
中島 僚佑
Deep Learningemi- upervised Learning

efined Consistencyによる知識蒸留を用いた半教師あり学習村本 佳隆
Deep Learning Reinforcement Learning Machine Learning

深層強化学習における価値・方策ベースの有効性の傾向調査
村上 遥
Deep Learning Echo tate Network Hessian-Free

Hessian-Free法を用いたEcho tate Networkの構造探索
濱地 優輝

URL: http://mprg.jp

ࢹઢ৘ใΛ༻͍ͨΞϯαϯϒϧਪ࿦ʹΑΔৄࡉը૾ࣝผͷߴਫ਼౓Խࢦಋ‫ڭ‬तɿ౻٢߂࿱

ER17078 ୩த ྄༎
1. ͸͡Ίʹ

ʹ෼ׂ͠ɼͦΕΒͷΫϥελͷ‫ۃ‬େҐஔΛऔಘ͢Δɽ

ৄࡉը૾ࣝผ͸ɼಉҰΧςΰϦ಺ʹ͓͍ͯɼࢹ֮తɾҙຯతʹྨࣅ͍ͯ͠Δ෺ମͷछྨΛࣝผ͢ΔλεΫͰ͋Δɽ
ৄࡉը૾ࣝผͷର৅Ϋϥε͸ɼਓ͕ؒೝࣝ͢Δ͜ͱ͑͞ࠔ೉ͳ΄Ͳࢹ֮తࠩҟ͕ͳ͍৔߹͕͋ΔɽͦͷͨΊɼৄࡉը૾ࣝผͰ͸ɼର৅෺ମͷશମͰ͸ͳ͘ɼಛ௃తͳྖҬΛଊ͑Δඞཁ͕͋Δɽͦ͜Ͱɼຊ‫Ͱڀݚ‬͸ɼࢹઢ৘ใΛ༻͍ͨΞϯαϯϒϧਪ࿦๏ʹ͍ͭͯఏҊ͢Δɽ

3.3 ࢹઢ৘ใΛ༻ֶ͍ͨशͱ TTA
ຊख๏Ͱ͸ɼࢹઢ৘ใΛ༻͍ͯΫϩοϓͨ͠ը૾Λֶशσʔλͱ͢Δɽͦͯ͠ɼਪ࿦Ͱ͸ਤ 2 ʹࣔ͢Α͏ʹɼࢹઢ৘ใΛ༻͍ͯ TTA Λߦ͏ɽຊख๏ͰͷΫϩοϓຕ਺͸ 4
ຕͱ͢ΔɽΞϯαϯϒϧਪ࿦͸ɼࣜ (1) ʹै͍ CNN ͷग़ྗ͔Βฏ‫ۉ‬Ϋϥε֬཰Λ‫ٻ‬ΊΔɽ"# $2. Ξϯαϯϒϧਪ࿦

3. ఏҊख๏
ຊ‫Ͱڀݚ‬͸ɼΞϯαϯϒϧਪ࿦࣌ʹਓؒͷࢹઢ৘ใΛར༻ͨ͠ TTA Λಋೖͨ͠ৄࡉը૾ࣝผख๏ΛఏҊ͢Δɽ

3.1 ࢹઢ৘ใʹΑΔαϯϓϦϯά
ৄࡉ෺ମࣝผʹ͓͍ͯɼਓͷ஌‫ݟ‬Λ༻͍ͯॏཁͱͳΔಛ௃ྖҬΛ༩͑Δ͜ͱ͸༗ޮͰ͋Δͱߟ͑Δɽͦ͜Ͱຊख๏
Ͱ͸ਓؒͷࢹઢ৘ใΛར༻͢Δɽ͋Β͔͡ΊΞΠτϥοΧʔΛ༻͍ͯσʔληοτͷ֤ը૾ʹର͠ɼࢹઢ৘ใΛऔಘ͢Δɽ࣍ʹɼಘΒΕͨࢹઢ৘ใ͔ΒαοέʔυΛআ֎͠ɼର
৅෺ମʹର͢Δը૾্ͷ஫໨࠲ඪσʔλΛ‫ٻ‬ΊΔɽֶश࣌‫ͼٴ‬ਪ࿦࣌ɼ஫໨࠲ඪσʔλʹର͠ Mean Shift ΫϥελϦϯάΛߦ͍஫ࢹ఺ΫϥελΛ‫ٻ‬ΊɼΫϥελͷ‫ۃ‬େҐஔΛΫϩοϓͷத৺Ґஔͱ͢Δɽ͜ΕΒͷΫϩοϓը૾Λ༻͍Δ͜ͱͰৄࡉ෺ମࣝผʹॏཁͱͳΔಛ௃ྖҬΛଊֶ͑ͨश‫ͼٴ‬ਪ࿦ΛՄೳʹ͢Δɽը૾ͷΫϩοϓ࣌ʹೖྗը૾֎
ʹ͸Έग़͢৔߹ɼྖҬ֎Λ [0,255] ͷ஋ͰຒΊΔ erasingɼ‫ྖͼٴ‬Ҭ಺ΛʹϦαΠζ͢Δ resize ͷ 2 छྨͷ͍ͣΕ͔Λߦ͏ɽ͜ΕʹΑΓɼը૾Ϋϩοϓ࣌ʹൃੜ͢ΔθϩύσΟ
ϯάʹΑΔਫ਼౓௿ԼΛ཈͑Δɽࢹઢ৘ใΛ༻͍ͨ‫఺ࢹݻ‬ͷີ౓ʹԠͨ͡ը૾ͷΫϩοϓΛਤ 1 ʹࣔ͢ɽ
erasing

アイトラッキング

Mean ShiftresizeConv5.x

Softmax

Conv5.x

Softmax

Conv5.x

Conv4.x
Conv4.x

Softmax

Conv3.x

Conv4.x

Conv1

Conv2.x

Conv3.x

Conv1

Conv2.x

Conv1

CNN

CNN

⊕

サンプリング$(1)

͜͜ͰɼM ͸σʔλ਺ɼpm ͸ωοτϫʔΫͷࣄ‫཰֬ޙ‬ɼx͸ೖྗσʔλΛද͢ɽҰൠ෺ମೝࣝͰ͸ɼϥϯμϜʹը૾ΛΫϩοϓͨ͠ΓίϯτϥετΛม͑ͨΓͯ͠ɼTTA Λߦ͏ɽ͔͠͠ɼৄࡉը૾ࣝผͷ৔߹ɼҟͳΔΫϥεؒͰಛ௃͕ྨࣅ͍ͯ͠Δ͜ͱ͕ଟ͘ɼϥϯμϜΫϩοϓʹΑΔ TTA͸ޮՌ͕௿͍ͱ͍͏໰୊͕͋Δɽ

Conv2.xM1 !
y=pm (x)
M m=1

CNN

Conv3.xΞϯαϯϒϧਪ࿦͸ෳ਺ͷग़ྗΛՃࢉ͠ฏ‫͢ۉ‬Δ͜ͱʹΑΓ൚ԽੑೳΛ޲্ͤ͞Δख๏Ͱ͋Δɽਪ࿦࣌ʹෳ਺ͷը૾Λೖྗ͠ɼͦΕΒͷ݁ՌΛ౷߹ͯ͠࠷ऴ݁ՌΛग़ྗ͢Δख๏͸ Test Time Augmentation(TTA) ͱ‫ݺ‬͹ΕΔɽTTA
ʹΑΔΞϯαϯϒϧਪ࿦͸ࣜ (1) ͷΑ͏ʹͳΔɽ

視線マップ

クロップ

CNN

%

アンサンブルCNN

ਤ 2: ࢹઢ৘ใΛ༻͍ͨ TTA ͷ֓ཁ

4. ධՁ࣮‫ݧ‬ఏҊख๏ͷ༗ޮੑΛ‫͢ূݕ‬ΔͨΊʹධՁ࣮‫ݧ‬Λߦ͏ɽ

4.1 ࣮‫ݧ‬৚݅ຊ࣮‫Ͱݧ‬͸ɼCaltech-UCSD Birds(CUB-200-2011)[1]σʔληοτʹɼࢹઢ৘ใΛ෇༩͠ɼ50 Ϋϥεɼ2,880 ຕΛ༻͍࣮ͯ‫ݧ‬Λߦ͏ɽֶश༻ʹ 2,015 ຕɼධՁ༻ʹ 865 ຕΛ༻͍Δɽը૾αΠζ͸ 256×256 ըૉʹϦαΠζͨ͠‫ޙ‬ɼࢹઢ৘ใΛ΋ͱʹ 112×112 ըૉʹΫϩοϓ͢ΔɽֶशϞσϧʹ͸ ResNet18 Λ࢖༻͢Δɽ

4.2 ࣮‫݁ݧ‬Ռ
ද 1 ʹࢹઢ৘ใͱϥϯμϜͳ TTA Λൺֱର৅ͱͨ݁͠
ՌΛࣔ͢ɽTTA ͷ֤݁Ռ͸ 5 ճࢼߦͨ͠ͱ͖ͷਖ਼ղ཰ͷฏ‫ۉ‬஋ͱඪ४ภࠩͰ͋Δɽද 1 ΑΓఏҊख๏͸ɼϥϯμϜͳ TTA ͱൺֱͯ͠ɼ1.71 ϙΠϯτਫ਼౓͕޲্ͨ͠ɽ
ද 1: ΫϩοϓαΠζมߋʹΑΔೝࣝਫ਼౓ [%]TTA ͳ͠
71.52

ϥϯμϜ

72.85 ± 0.93

TTA ͋Γ
ࢹઢ৘ใ
erasing
resize74.56 ± 0.73

73.86 ± 0.71·ͨɼϥϯμϜͱࢹઢ৘ใΛ༻͍ͨ৔߹ͷֶश‫ۂ‬ઢΛਤ
3 ʹࣔ͢ɽਤ 3 ΑΓɼࢹઢ৘ใΛ༻͍ͨఏҊख๏͸ֶशͷૣ‫Ͱظ‬ऩଋ͍ͯ͠Δɽ͜ΕΑΓɼఏҊख๏͸গͳ͍ΤϙοΫͰΫϥεಛ༗ͷৄࡉͳಛ௃Λޮ཰తʹֶशͰ͖͍ͯΔ͜
ͱ͕Θ͔Δɽ

クロップ

ਤ 1: ‫఺ࢹݻ‬ͷີ౓ʹԠͨ͡ը૾ͷΫϩοϓ

(a) ଛࣦͷਪҠ

3.2 Mean Shift ΫϥελϦϯάMean Shift ͱ͸ɼۙ๣ͷ఺‫܈‬ͷฏ‫ۉ‬ҐஔʹҠಈΛ‫܁‬Γฦ͠ɼू߹ͷ‫ۃ‬େ఺Λ‫ٻ‬ΊΔํ๏Ͱ͋Δɽ‫܁‬Γฦ͠εςοϓʹ͓͚Δ‫ۃ‬େ఺୳ࡧΛࣜ (2) ʹࣔ͢ɽ"nK(xi ; x, h)xi
s(x) = m(x) − x = "i=1−x(2)n
i=1 K(xi ; x, h)͜͜ͰɼK ͸Χʔωϧؔ਺ɼxi ͸఺ͷू߹ɼh ͸όϯυ෯Λද͠ɽs(x) ͸‫ۃ‬େ఺ʹۙͮ͘΄Ͳ஋͕খ͘͞ͳΔɽ֤఺xi Λऩଋઌͷ‫ۃ‬େ఺͝ͱʹϥϕϧ෇͚͢Δ͜ͱͰΫϥελ(b) ೝࣝ཰ͷਪҠਤ 3: ֶश‫ۂ‬ઢ

5. ͓ΘΓʹ
ຊ‫Ͱڀݚ‬͸ɼࢹઢ৘ใΛ༻͍ͨΞϯαϯϒϧʹΑΔৄࡉ
෺ମࣝผ๏ΛఏҊͨ͠ɽࠓ‫ޙ‬ͷల๬ͱͯ͠ɼࢹઢਪఆϞσϧΛ༻͍ͨҰ؏ֶशʹΑΔ TTA Λ‫ݕ‬౼͢Δɽ

ࢀߟจ‫ݙ‬[1] Wah. C, et al. “The Caltech-UCSD Birds-200-2011
Dataset”, Technical Report CNS-TR-2011-001, Caltech,2011.

2 ஈ֊൑ఆʹΑΔΦΫϧʔδϣϯʹରԠͨ͠าߦऀ‫ݕ‬ग़ࢦಋ‫ڭ‬तɿ౻٢ ߂࿱

ER17001 ळࢁ ‫ݡ‬थ
1. ͸͡Ίʹ

2. ؔ࿈‫ڀݚ‬࠷ऴ൑ఆΛߦ͏ 2 ஈ֊ͷาߦऀ‫ݕ‬ग़ख๏ΛఏҊ͢ΔɽఏҊख๏ͷྲྀΕΛਤ 3 ʹࣔ͢ɽ1 ஈ֊໨Ͱ͸ɼೖྗը૾Λ CSP
ʹೖྗ͠ɼาߦऀͷ֬৴౓ϚοϓΛ‫ٻ‬ΊΔɽ͜ͷͱ͖ɼ֬৴౓ͷϚοϓʹରͯ͠ɼᮢ஋Λ௿͘ઃఆͯ͠ଟ͘ͷީิྖҬΛग़ྗ͢Δɽ࣍ͷɼ2 ஈ֊໨Ͱ͸ɼาߦऀީิྖҬΛ 1
ͭͣͭ Compositional Nets ʹೖྗ͠ɼީิྖҬͷΫϥε֬཰ͱΦΫϧʔδϣϯϚοϓΛग़ྗ͢ΔɽΫϥε֬཰Λ‫ج‬ʹީิྖҬΛาߦऀ͔൱͔Λ൑ఆ͢Δɽ

Input image

Detection Head
Heatmap：Deconv.

ຊ࣮‫Ͱݧ‬͸ɼैདྷͷ CSP ͱఏҊख๏Λൺֱ͠ɼͦͷ༗ޮੑΛධՁ͢Δɽ

4.1 ࣮‫֓ݧ‬ཁֶशɼධՁʹ͸ CityPersons σʔληοτΛ༻͍ΔɽධՁࢦඪʹ͸ Miss Rate(ະ‫ݕ‬ग़཰) Λ༻͍ΔɽධՁ͸ɼBareɼ
PartialɼHeavyɼReasonable ͷΦΫϧʔδϣϯ౓߹͝ͱʹ Miss Rate Λࢉग़͠ɼैདྷख๏ (CSP ୯ମ) ͱͷൺֱΛߦ͏ɽैདྷख๏ͱఏҊख๏Λൺֱͨ݁͠ՌΛද 1 ʹࣔ͢ɽද 1ΑΓɼఏҊख๏Λैདྷख๏ͱൺֱͯ͠ɼHeavy ʹ͓͍͖͍ͯ͠஋ 0.25 ͱͨ࣌͠ɼMiss Rate ͷਫ਼౓Λ 6.76 ϙΠϯτ޲্ͤ͞Δ͜ͱ͕ग़དྷͨɽ ఏҊख๏ͷ‫ݕ‬ग़݁ՌͱΦΫϧʔOffsets：Conv.

4. ධՁ࣮‫ݧ‬

4.2 ࣮‫݁ݧ‬Ռ

Scale Map

L2 Norm

Concatenate

L2 Norm

：歩⾏者

2段階⽬

1段階⽬

ਤ 3: ఏҊख๏ͷ 2 ஈ֊൑ఆCSP ͷωοτϫʔΫߏ଄Λਤ 1 ʹࣔ͢ɽCSP ͸าߦऀͷத৺఺ͱॎԣͷεέʔϧΛώʔτϚοϓͰਪఆ͢Δ
keypoint ‫ܕ‬ͷ‫ݕ‬ग़ख๏Ͱ͋ΔɽώʔτϚοϓ಺ͷ‫ۃ‬େ஋Λาߦऀͷத৺఺ͱ͠ɼ‫ۃ‬େ஋͕าߦऀͷ֬৴౓ͱͳΔɽ
Center and Scale Prediction [W. Liu+, CVPR2019]
CSP ͸ΦΫϧʔδϣϯ͕ൃੜ͍ͯͯ͠΋Մࢹ෦෼ͷาߦऀ• 歩⾏者の中⼼位置とそこからの縦横のスケールを推定ͷಛ௃͔Βத৺ҐஔΛಛఆ͠ɼ‫ݕ‬ग़ྖҬͷਪఆ͕ՄೳͰ͋
– Keypoint型物体検出⼿法
Δɽ͔͠͠ͳ͕ΒɼΦΫϧʔδϣϯ͕ൃੜ͍ͯ͠Δาߦऀʹର͢Δ֬৴౓͸௿͘ɼ‫ݕ‬ग़Ͱ͖ͳ͍έʔε΋͋Δɽ

L2 Norm

Compositional Nets

Compositional Nets

2.1 Center and Scale Prediction

L2 Norm

Compositional Nets

Heatmapຊ‫͍༻Ͱڀݚ‬ΔɼCSP ͱ Compositional Nets ͷߏ଄ʹ͍ͭͯड़΂Δɽ

Feature Extraction

検出結果 隠れマップ

候補領域

候補領域判定(閾値処理)

CSP

าߦऀ‫ݕ‬ग़͸ɼը૾಺͔ΒาߦऀͷҐஔͱେ͖͞Λਪఆ
͢Δٕज़Ͱ͋Γɼࣗಈӡసʹ͓͍ͯॏཁͳ‫ػ‬ೳͱͳΔɽาߦऀ‫ݕ‬ग़ͷ୅දతͳख๏Ͱ͋ΔɼCenter and Scale Prediction(CSP)[1] ͸ɼConvolutional Neural Network(CNN)Λϕʔεͱ͠ɼߴਫ਼౓ʹาߦऀͷத৺఺ͱεέʔϧΛਪఆ͢ΔɽҰํͰɼࣗಈӡసͰ͸ɼาߦऀͷҰ෦ʹΦΫϧʔδϣϯ͕ੜ͍ͯ͡ΔέʔεͰ΋ਖ਼͘͠‫ݕ‬ग़͢Δඞཁ͕͋Δɽ
CSP ͸ΦΫϧʔδϣϯ͕ൃੜ͢Δͱɼาߦऀͷ֬৴౓͕௿
Լ͠ɼະ‫ݕ‬ग़ͱͳΔɽͦ͜Ͱຊ‫Ͱڀݚ‬͸ɼCSP Λ༻͍ͯาߦऀީิྖҬΛ‫ݕ‬ग़ͨ͠‫ޙ‬ɼCompositional Nets[2] ʹΑΔΦΫϧʔδϣϯΛߟྀͨ͠࠷ऴ൑ఆΛߦ͏ 2 ஈ֊ͷาߦऀ‫ݕ‬ग़ख๏ΛఏҊ͢Δɽද 1: ࣮‫݁ݧ‬Ռ (Miss Rate)3ਤ 1: Center and Scale Prediction
2.2 Compositional Nets
Compositional Nets ͸ΦΫϧʔδϣϯ͕ൃੜ͍ͯ͠Δ
౓߹ΛώʔτϚοϓͱͯ͠ग़ྗ͢Δɽ·ͨɼ෺ମΒ͠͞΋߹ͤͯग़ྗ͢ΔɽͦͷͨΊɼ෺ମͷ൑ఆͱΦΫϧʔδϣϯൃੜͷ౓߹Λಉ࣌ʹଊ͑Δ͜ͱ͕Ͱ͖ΔɽCompositional
Nets ͸ DCNN Ͱ৞ΈࠐΈॲཧΛߦ͍ɼը૾ͷಛ௃Λநग़
͢Δɽͦͯ͠ɼநग़ͨ͠ಛ௃ϚοϓΛ vMFKarnels ʹೖྗ͠ಛ௃ϕΫτϧʹ֬཰Λ෇༩͢ΔɼOccluder kernels Ͱ͸ɼͦͷ֬཰Λ‫ʹج‬ΦΫϧʔδϣϯ෦෼ͱͦΕҎ֎ͷ෦෼ʹ෼ྨͨ͠ΦΫϧʔδϣϯϚοϓΛग़ྗ͢ΔɽClass mixturesͰ͸ɼίϯςΩετ৘ใͱΦϒδΣΫτ৘ใΛ߹Θͤͨ෺ମϚοϓΛग़ྗ͢ΔɽΦΫϧʔδϣϯϚοϓͱ෺ମϚοϓΛ౷߹͠ɼ࠷ऴతͳΫϥεείΞͱΦΫϧʔδϣϯϚοϓΛग़ྗ͢Δɽ

ैདྷ๏
ఏҊख๏

ᮢ஋

Reasonable

Bare

Partial

Heavy

0.5

20.28

15.49

21.03

60.90

0.1

17.93

13.30

18.03

55.55

0.25

16.82

12.37

16.48

54.14

0.5

18.65

13.96

19.10

58.65

δϣϯϚοϓΛਤ 4 ʹࣔ͢ɽਤ 4 ͔ΒఏҊख๏͸ɼาߦऀͷΦΫϧʔδϣϯྖҬͷਪఆʹ੒ޭ͠ɼਖ਼͍͠൑ఆ݁ՌΛ
ग़ྗ͢Δ͜ͱ͕Ͱ͖ɼ‫ݕ‬ग़Ͱ͖͍ͯΔ͜ͱ͕෼͔Δɽ(a) ैདྷ๏(b) ఏҊख๏(c) ӅΕϚοϓ

ਤ 4: ఆੑత݁Ռ

5. ͓ΘΓʹ
ຊ‫Ͱڀݚ‬͸ධՁ࣮‫͔ݧ‬ΒΦΫϧʔδϣϯͷߟྀΛͨ͠ 2 ஈ֊ͷ൑அʹΑΓਫ਼౓͕޲্͢Δ͜ͱΛ֬ೝͨ͠ɽࠓ‫ޙ‬͸ɼຊఏҊख๏ͷߋͳΔߴਫ਼౓ԽΛ໨ࢦ͢ɽࢀߟจ‫ݙ‬
ਤ 2: Compositional Nets

3. ఏҊख๏
ຊ‫Ͱڀݚ‬͸ɼCSP Λ༻͍ͯาߦऀީิྖҬΛ‫ݕ‬ग़ͨ͠‫ޙ‬ɼCompositional Nets ʹΑΔΦΫϧʔδϣϯΛߟྀͨ͠[1] W. Liu et al.,“Center and Scale Prediction:A Box -free
Approach for Pedestrain and Face Detection”ɼ CVPR
2019.
[2] A. Kortylewsk, et al., “Compositional Convolutional
Neural Networks:A Deep Architecture with Innate Robustness to Partial Occlusion”ɼ CVPR 2020.

Mixup ֦ுʹର͢Δ ABN ʹΑΔΞςϯγϣϯͷ֫ಘͱೝࣝਫ਼౓΁ͷӨ‫ڹ‬ࢦಋ‫ڭ‬तɿ౻٢߂࿱
ER17004 Ո࡚ ߊ
1. ͸͡Ίʹ
ਂ૚ֶशͷϗϫΠτϘοΫεԽͱͯ͠ɼೝࣝ࣌ʹ൑அࠜ‫ڌ‬ΛՄࢹԽ͠ࢹ֮తઆ໌Λߦ͏͜ͱ͸ॏཁͰ͋ΔɽAttention
Branch Network(ABN)[1] ͸ɼࢹ֮తઆ໌͚ͩͰ͸ͳ͘ɼ൑அࠜ‫ڌ‬ΛΞςϯγϣϯͱͯ͠Ξςϯγϣϯ‫ʹߏػ‬Ԡ༻͢
Δ͜ͱͰՄࢹԽʹߩ‫͍ͯ͠ݙ‬ΔɽҰํͰɼ෺ମೝࣝͰ͸ֶशσʔλͱͦͷόϦΤʔγϣϯΛ֬อ͢ΔͨΊʹը૾߹੒ͯ͠ਫ૿͢͠Δσʔλ֦ு͕‫͞༻׆‬Ε͍ͯΔɽಛʹը૾Λ߹੒͢ΔΞϓϩʔν͸ਫ਼౓޲্ʹେ͖͘‫د‬༩͍ͯ͠Δɽ͠
͔͠ͳ͕Β ABN ʹ͓͍ͯɼը૾߹੒Λߦ͏σʔλ֦ுͷӨ‫े͕ڹ‬෼ʹߟྀ͞Ε͍ͯͳ͍ɽͦ͜Ͱɼຊ‫Ͱڀݚ‬͸ը૾߹੒Λߦ͏σʔλ֦ுΛ௥Ճ͢Δ͜ͱͰɼABN ͷ൑அࠜ‫ڌ‬͓Αͼਫ਼౓ʹରͯ͠ͲͷΑ͏ͳӨ‫͕ڹ‬ग़Δͷ͔Λௐࠪ͢Δɽ

2.Attention Branch Network(ABN)ABN ͸ɼ෺ମೝࣝ࣌ͷ൑அࠜ‫ͳͱڌ‬Δ஫໨ྖҬΛΞςϯγϣϯϚοϓͱͯ͠ՄࢹԽͭͭ͠ɼೝࣝʹར༻͢Δख๏Ͱ
͋Δɽਤ 1 ͷΑ͏ʹ ABN ͸ɼFeature extractorɼAttention branchɼPerception branch ͷ̏ͭͷϞδϡʔϧ͔Βߏ੒͞Ε͍ͯΔɽFeature extractor ͸ɼೖྗը૾ʹର͢Δ
ಛ௃ϚοϓΛநग़͢Δɽநग़ͨ͠ಛ௃ϚοϓΛ Attention
branch ʹ༩͑ɼΞςϯγϣϯϚοϓΛ֫ಘ͢ΔɽΞςϯγϣϯϚοϓ͸ೖྗը૾ʹର͢Δ஫ࢹྖҬΛࣔ͠ɼFeature
extractor ͷಛ௃Ϛοϓʹ৐ࢉ͠ɼPerception branch ʹ༩͑ΔɽPerception branch ͸ೝࣝ݁ՌΛग़ྗ͢Δɽ

श܎਺Λ 0.1 ʹઃఆ͠ɼCIFAR100 Ͱ͸ 150 ͱ 225epochɼ
ImageNet Ͱ͸ 100 ͱ 150epoch ͷ࣌ʹֶश܎਺Λ 0.1 ഒ͢ΔɽTop1 ਫ਼౓ʹՃ͑ɼ൑அࠜ‫ڌ‬ͷఆྔతධՁʹ͸ insertion
ͱ deletion Λ༻͍Δɽinsertion ͸ॏཁͳྖҬΛ௥Ճ͍ͯ͠
͖ɼͦͷ࣌ͷೝࣝਫ਼౓͕Θ͔ͣͳྖҬͰ΋޲্͢Δͱྑ͍ɽ
deletion ͸ॏཁͳྖҬΛ࡟আ͍͖ͯ͠ɼͦͷ࣌ͷೝࣝਫ਼౓͕ྖҬͰ΋௿Լ͢Δͱྑ͍ɽ͜ΕΒͷྖҬΛ௥Ճ·ͨ͸࡟আ͢Δׂ߹Λม͑ɼͦͷ࣌ͷ Area Under Curve (AUC)
ͰධՁ͢Δɽ

4.2. ࣮‫݁ݧ‬Ռ
CIFAR100 ͱ ImageNet ͰͷධՁ݁ՌΛද 1 ʹࣔ͢ɽCIFAR100 Ͱ͸ɼResNet20 ͱ Resnet152 ͲͪΒ΋σʔλ֦ுΛ༻͍Δ͜ͱͰਫ਼౓͕޲্͍ͯ͠Δ͜ͱ͕֬ೝͰ͖Δɽ·ͨɼωοτϫʔΫϞσϧʹΑͬͯ࠷΋ੑೳͷྑ͍σʔλ֦ுख๏͕ҟͳΔ͜ͱ͕Θ͔ͬͨɽImageNet Ͱ͸ɼResNet18
ʹ͓͍ͯ cutout ͷਫ਼౓޲্͕‫ݟ‬ΒΕ͕ͨɼmixup ͱ cutmix Ͱ͸ਫ਼౓ͷ޲্͕‫ݟ‬ΒΕͳ͔ͬͨɽResNet152 ʹ͓͍ͯ͸ɼͲͷσʔλ֦ுΛ༻͍ͯ΋ਫ਼౓͕޲্͍ͯ͠Δ͜ͱ
͕֬ೝͰ͖ͨɽ͜ΕΒͷ݁Ռ͔Βɼେ‫ن‬໛ͳσʔληοτΛର৅ͱ͢Δ৔߹ωοτϫʔΫαΠζͷখ͍͞Ϟσϧʹରͯ͠͸ɼը૾σʔλͷϥϕϧΛม͑ͳ͍ cutout ͕༗ޮతͰ͋ΓɼωοτϫʔΫαΠζͷେ͖͍Ϟσϧʹରͯ͠͸ɼ2ຕͷը૾Λ߹੒͢ΔΑ͏ͳ mixup ΍ cutmix ͕༗ޮͰ͋Δ
ͱߟ͑ΒΕΔɽɹද 1: ࣮‫݁ݧ‬ՌAttention mapσʔληοτ

Input

Attentionmechanism

Perception branch

ImageNetResNet152

ResNet18

ResNet152

DA ͳ͠

69.32

76.78

70.34

77.87

+mixup

70.19

79.71

67.64

77.90

𝐿𝑎𝑡𝑡

+cutout

72.58

78.79

70.87

78.74

+cutmix

71.05

80.80

68.07

79.32

Attention branch

Featureextractor

CIFAR100ResNet20

LabelωοτϫʔΫ

𝐿𝑝𝑒𝑟ਤ 1: Attention Branch Network ͷߏ੒

3. σʔλ֦ுσʔλ֦ுʹ͸ɼฏߦҠಈɾ֦େॖখͳͲͷ‫ز‬Կֶม‫׵‬ΛՃ͑Δํ๏ɼ໌Δ͞΍ϊΠζΛՃ͑Δํ๏ɼը૾Λ߹੒
͢Δํ๏͕͋Δɽը૾Λ߹੒͢Δํ๏ʹ͸ɼ2 ͭͷը૾ͱϥϕϧΛࠞ߹͢Δ mixupɼը૾಺ͷ͋ΔྖҬΛϥϯμϜʹ
ࠇ͘ృΓͭͿ͢ cutoutɼը૾ͷҰ෦ʹϥϯμϜͳը૾Λ͸ΊࠐΉ cutmix ͕͋Δɽ߹੒ͨ͠σʔλΛֶश༻σʔλʹ
௥Ճ͢Δ͜ͱͰաֶशΛ཈੍Ͱ͖ɼੑೳͷ޲্͕‫ظ‬଴Ͱ͖
Δɽਤ 2 ʹ֤ख๏ͷྫΛࣔ͢ɽ

ImageNet ͰͷΞςϯγϣϯϚοϓͷՄࢹԽ݁ՌΛਤ 3ʹࣔ͢ɽਤ 3 ΑΓɼσʔλ֦ுΛ༻͍Δ͜ͱͰਫ਼౓͸޲্͕ͨ͠ɼmixup ͱ cutmix Ͱ͸஫ࢹ͢ΔྖҬ͕૿͍͑ͯΔɽҰํɼcutout Ͱ͸஫ࢹྖҬ͕‫ݶ‬ఆ͞Ε͍ͯΔɽ
ɹΞςϯγϣϯϚοϓͷධՁ݁ՌΛද 2 ʹࣔ͢ɽද 2 ΑΓɼσʔλ֦ுલͱൺֱͯ͠ਫ਼౓޲্͸‫ݟ‬ΒΕͳ͍ɽ͜ΕΑΓɼ
mixup ϕʔεͷσʔλ֦ு͸ɼਫ਼౓޲্ʹߩ‫͢ݙ‬Δ͕ɼΞςϯγϣϯϚοϓʹѱӨ‫ڹ‬Λ‫͕ͱ͜͢΅ٴ‬෼͔ͬͨ.

(a) ը૾

(b)ResNet152(c)+mixup (d)+cutout (e)+cutmix

ਤ 3: ΞςϯγϣϯϚοϓͷൺֱද 2: insertion ͱ deletion ʹΑΔఆྔతධՁ (AUC)(a) mixup(b) cutout(c) cutmix

ਤ 2: σʔλ֦ுͷྫ

4. ௐࠪ
ABN ͸ɼΞςϯγϣϯϚοϓΛಛ௃Ϛοϓʹ൓өͤ͞
Δ͜ͱͰೝࣝʹॏཁͱͳΔྖҬʹ஫໨ͨ͠ೝࣝ݁ՌͱͳΔɽͦͷͨΊɼσʔλ֦ுʹΑΓ߹੒͞Εͨը૾Λ༻͍Δͱɼ
ΞςϯγϣϯϚοϓʹѱӨ‫ڹ‬Λ‫͑ߟ͕ͱ͜͢΅ٴ‬ΒΕΔɽ
ຊ‫Ͱڀݚ‬͸ɼmixupɼcutoutɼcutmix ͷ 3 ͭͷσʔλ֦ுख๏Λ ABN ͷֶशʹ༻͍ͯɼͦΕΒͷӨ‫ڹ‬Λௐࠪ͢Δɽ

4.1. ࣮‫ݧ‬৚݅ຊ࣮‫Ͱݧ‬͸ɼσʔληοτʹ CIFAR100 ͱ ImageNetΛ༻͍Δɽֶशύϥϝʔλ͸όοναΠζΛ 128ɼepoch਺Λ CIFAR100 Ͱ͸ 300ɼImageNet Ͱ͸ 200 ͱ͢Δɽֶ

ωοτϫʔΫ

insertion ˢdeletion ˣ

ResNet152

0.267

0.426

+ mixup

0.224

0.452

+ cutout

0.261

0.451

+ cutmix

0.271

0.474

5. ͓ΘΓʹ
ࠓճ͸ɼABN ʹσʔλ֦ுΛ༻͍ͨࡍͷਫ਼౓ൺֱΛߦͬͨɽωοτϫʔΫαΠζͷมԽʹΑΓɼ࠷΋ੑೳͷྑ͍σʔλ֦ுख๏͕ҟͳΔͱ͍͏͜ͱ͕෼͔ͬͨɽࠓ‫ޙ‬͸ɼ͜ΕΒͷσʔλ֦ுख๏Λ૊Έ߹ΘͤͨࡍͷӨ‫͍ͯͭʹڹ‬ௐࠪ͢Δɽࢀߟจ‫ݙ‬[1] H. Fukui, et al., “Attention Branch Network: Learning of Attention Mechanism for Visual Explanation”, CVPR, 2019.

Attention mining branch Λಋೖͨ͠ ABN ʹΑΔΞςϯγϣϯϚοϓͷ࠷దԽࢦಋ‫ڭ‬तɿ౻٢ ߂࿱
ER17009 ‫ؠ‬٢ ޹໌
1. ͸͡Ίʹ

4. ධՁ࣮‫ݧ‬

Attention Branch Network (ABN)[1] ͸ɼೝࣝ࣌ʹ͓
͚ΔωοτϫʔΫͷ஫ࢹྖҬΛՄࢹԽ͠ɼೝࣝॲཧʹ‫༻׆‬
͢Δ͜ͱͰɼߴਫ਼౓ԽΛ࣮‫͍ͯ͠ݱ‬Δɽೝࣝର৅ͷ෺ମҎ֎ʹ஫ࢹྖҬ͕ൃੜ͢Δͱɼ‫ޡ‬ೝࣝΛ༠ൃ͢Δ͜ͱ͕͋Δɽ͜ΕΛղܾ͢Δख๏ͱͯ͠ɼਓͷ஌‫ʹݟ‬ΑΓमਖ਼ͨ͠ΞςϯγϣϯϚοϓΛ༻͍ͯ ABN Λ࠶ֶश͢Δख๏ [2] ͕ఏҊ͞Ε͍ͯΔɽ͔͠͠ͳ͕Βɼຊख๏͸ΞςϯγϣϯϚο
ϓΛਓ͕मਖ਼͢ΔͨΊਓతίετֻ͕͔Δɽຊ‫Ͱڀݚ‬͸ɼ
ABN ʹ Attention mining branch Λಋೖ͠ɼΞςϯγϣ
ϯϚοϓΛࣗಈͰ࠷దԽ͢Δख๏ΛఏҊ͢Δɽ

2.Attention Branch Network
ABN ͸ɼFeature extractorɼAttention branchɼPerception branch ͔Βߏ੒͞ΕΔɽFeature extractor ͸ೖྗը૾͔Βಛ௃ϚοϓΛநग़͢Δɽ͜ΕΛ Attention branchʹ༩͑ɼΞςϯγϣϯϚοϓΛ֫ಘ͢ΔɽΞςϯγϣϯϚο
ϓΛಛ௃Ϛοϓʹ৐ࢉ͠ɼPerception branch ʹ༩͑Δ͜
ͱͰೝࣝ݁ՌΛग़ྗ͢ΔɽABN ΛԠ༻ͨ͠‫ͯ͠ͱڀݚ‬ɼਓͷ஌‫ݟ‬Λಋೖ͢Δख๏͕͋Δɽ͜ͷख๏Ͱ͸ɼABN Ͱ‫ޡ‬ೝ͕ࣝൃੜͨ͠ը૾ͷΞςϯγϣϯϚοϓΛਓͷ஌‫جʹݟ‬͍ͮͯ͢΂ͯमਖ਼͢Δɽͦͷ‫ޙ‬ɼͦΕΒΛֶशσʔλʹ௥Ճͯ͠࠶ֶश͢Δ͜ͱΑΓೝࣝਫ਼౓͕޲্͢ΔɽఏҊख๏ͷ༗ޮੑΛධՁ͢ΔͨΊʹධՁ࣮‫ݧ‬Λߦ͏ɽ

4.1 ࣮‫ݧ‬৚݅ຊ࣮‫Ͱݧ‬͸ɼCaltech-UCSD Birds 200-2010 (CUB-2002010) σʔληοτͱɼStanford Dogs σʔληοτΛ༻͍ΔɽϕʔεωοτϫʔΫͱͯ͠ ResNet-50 Λ࢖༻͠ɼόοναΠζ͸ 16 ͱ͢ΔɽMask ͷᮢ஋͸ CUB-200-2010 σʔληοτͰ 0.83ɼStanford Dogs σʔληοτͰ 0.40 ͱ͢ΔɽLam ͷ܎਺ α ͸ 0.0001 ʹઃఆ͢Δɽֶशͷߋ৽ճ਺͸ ABN ͷࣄલֶशɼఏҊख๏ͦΕͧΕͰ 300epoch ͱ͢Δɽ

4.2 ࣮‫݁ݧ‬Ռ֤σʔληοτʹ͓͚Δೝࣝਫ਼౓ͷൺֱΛද 1 ʹࣔ͢ɽ
ද 1 ΑΓɼCUB-200-2010 σʔληοτʹ͓͍ͯɼఏҊख๏͸ɼਓͷ஌‫ݟ‬Λಋೖ͢Δख๏ΑΓ΋ Top-1 ͷೝࣝਫ਼౓͕
௿͘ͳ͍ͬͯΔ͕ɼABN ΑΓ΋޲্͍ͯ͠Δ͜ͱ͕֬ೝ
Ͱ͖Δɽ·ͨɼStanford Dogs σʔληοτʹ͓͍ͯɼఏ
Ҋख๏͕ ABN ΑΓ΋ɼTop-1 ͷೝࣝਫ਼౓͕޲্͍ͯ͠Δ
͜ͱ͕֬ೝͰ͖Δɽ
ද 1 : ೝࣝਫ਼౓ͷൺֱ [ˋ]Attention mechanism

Perception
branch：binarizationTop-5 acc.

Top-1 acc.

Top-5 acc.

ABN

31.68

57.01

71.81

93.02

ఏҊख๏

33.33

58.56

71.99

92.80

ਓͷ஌‫ݟ‬

37.42

62.08

-

-

Input

weightshare

Attention
module4.3 ΞςϯγϣϯϚοϓͷՄࢹԽ
ABNɼఏҊख๏ɼਓͷ஌‫ݟ‬Λಋೖ͢Δख๏ͷ Attention
map ͷൺֱΛਤ 2 ʹࣔ͢ɽਤ 2 (a)ɼ(b) ΑΓɼਓͷ஌‫ݟ‬Λಋೖ͢Δख๏͸ɼΑΓ‫ॴہ‬తͳྖҬʹண໨ͯ͠Ϋϥεࣝผ͕Ͱ͖͍ͯΔɽҰํͰɼਤ 2 (c) ͷΑ͏ʹɼਓͷ஌‫ݟ‬Λ൓өͤ͗͢͞Δͱ‫ޡ‬ೝࣝΛ༠ൃ͢Δࣄ͕͋Δɽ͜Εʹର͠ɼఏҊख๏͸޿ൣғΛ஫ࢹͰ͖͍ͯΔͨΊਖ਼͘͠ೝ͍ࣝͯ͠Δɽ

Mask

Featureextractor

Stanford Dogs

Top-1 acc.

3. ఏҊख๏
ຊ‫Ͱڀݚ‬͸ɼABN ʹ Attention mining branch Λಋೖͨ͠ΞςϯγϣϯϚοϓͷ࠷దԽख๏ΛఏҊ͢ΔɽఏҊख๏ͷωοτϫʔΫߏ଄Λਤ 1 ʹࣔ͢ɽ

CUB-200-2010

Model

Attention map

Attentionmining branchਤ 1 : ఏҊख๏ఏҊख๏Ͱ͸ɼABN ʹ Perception branch ͱॏΈΛ‫ڞ‬༗ͨ͠ Attention mining branch Λ௥Ճ͢ΔɽAttention
mining branch ʹ͸ɼMask Λద༻ͨ͠ಛ௃ϚοϓΛೖྗ͠ɼΫϥε֬཰Λग़ྗ͢Δɽ͜͜ͰɼMask ͸ Attention
module Ͱ֫ಘͨ͠ΞςϯγϣϯϚοϓΛ 2 ஋Խͯ͠ੜ੒
͢Δɽ͜ͷ Mask Λಛ௃Ϛοϓʹ৐ࢉ͢Δ͜ͱͰ஫ࢹͨ͠
ྖҬΛӅͨ͠ಛ௃ϚοϓΛ֫ಘͰ͖ΔɽMask ద༻‫ޙ‬ͷಛ௃Ϛοϓ F ∗c ͸ɼಛ௃ϚοϓΛ F ɼMask Λ T (Ac ) ͱ͢Δͱࣜ (1) ͷΑ͏ʹද͞ΕΔɽ

F ∗c = F − (T (Ac ) ⊙ F )(1)ఏҊख๏Ͱ͸ɼAttention mining branch ͔Βग़ྗ֤ͨ͠
αϯϓϧͷΫϥε֬཰ͷ૯࿨Λ৽ͨͳଛࣦͱ͢Δɽ͜ΕʹΑΓɼMask Λద༻͍ͯ͠ͳ͍ྖҬͷΫϥε֬཰Λ࠷খԽ͢ΔΑ͏ʹֶश͠ɼೝࣝର৅ͷ෺ମͷΈΛ஫ࢹ͢ΔΑ͏ʹΞςϯγϣϯϚοϓΛ࠷దԽ͢ΔɽAttention mining branchͷଛࣦ Lam ͸ɼΫϥε֬཰Λ Sic (F ∗c )ɼαϯϓϧ਺Λ nͱ͢Δͱࣜ (2) ͷΑ͏ʹද͞ΕΔɽ
n
!
Lam =
Sic (F ∗c )(2)
i=1

Attention module ͷग़ྗͱਖ਼ղΫϥεͷΫϩεΤϯτϩ
ϐʔ‫ࠩޡ‬Λ Latt ɼPerception branch ͷग़ྗͱਖ਼ղΫϥεͷΫϩεΤϯτϩϐʔ‫ࠩޡ‬Λ Lper ɼLam ͷॏΈΛ α ͱ͢ΔͱɼఏҊख๏ͷଛࣦؔ਺ L ͸ࣜ (3) ͷΑ͏ʹද͞ΕΔɽ

(a)

(b)

(c)

ਤ 2 : ΞςϯγϣϯϚοϓͷՄࢹԽྫ

5. ͓ΘΓʹ
ຊ‫Ͱڀݚ‬͸ɼAttention mining branch Λ௥Ճͨ͠ ABN
ΛఏҊͨ͠ɽධՁ࣮‫Ͱݧ‬͸ɼఏҊख๏Λ༻͍Δ͜ͱʹΑΓɼೝࣝਫ਼౓͕޲্ͨ͠ɽ·ͨɼ஫ࢹྖҬʹ͍ͭͯ΋վળ͞Ε
͍ͯΔ͜ͱ͕֬ೝͰ͖ͨɽࠓ‫ޙ‬͸ɼଞͷσʔληοτʹରͯ͠ఏҊख๏ͷධՁΛߦ͏ɽࢀߟจ‫ݙ‬(3)[1] H. Fukui, et al., “Attention Branch Network:Learning ofAttention Mechanism for Visual Explanation”, CVPR,
2019.

ఏҊख๏Ͱ͸ɼஈ֊తʹֶश͢Δɽ·ͣ ABN ͷΈΛࣄલ
ֶश͢Δɽͦͯ͠ɼAttention mining branch Λ௥Ճͯ͠ΞςϯγϣϯϚοϓͷ࠷దԽͱͳΔ࠶ֶशΛߦ͏ɽ

[2] M. Mitsuhara, et al., “Embedding Human Knowledge
into Deep Neural Network via Attention Map”, VISAPP,
2021.

L = Latt + Lper + αLam

Attention branch ͷಗ໊ԽʹΑΔ Adversarial examples ʹ‫ ͳ݈ؤ‬ABN
ࢦಋ‫ڭ‬तɿ౻٢߂࿱
ER17079 ࢁ઒ᠳଠ࿠
1. ͸͡Ίʹ

4.1. ࣮‫ݧ‬৚݅Attention branch network (ABN) [1] ͸ɼֶश࣌·ͨ͸ਪ࿦࣌ʹωοτϫʔΫͷ஫ࢹྖҬΛར༻͢Δ͜ͱͰɼߴਫ਼౓
ͳೝ͔ࣝͭ൑அࠜ‫ڌ‬ͷࢹ֮తͳઆ໌͕ՄೳͰ͋Δɽ͔͠͠ɼ
ਓ޻తͳઁಈΛ෇༩ͨ͠ը૾Ͱ͋Δ Adversarial examples(AEs) ΛωοτϫʔΫ΁ೖྗ͢Δͱɼߴ͍৴པ౓Ͱ‫ޡ‬ೝࣝ͢Δɽͦ͜Ͱɼຊ‫Ͱڀݚ‬͸ ABN ͷ Attention branch Λಗ໊Խ͢Δ͜ͱʹΑΓɼAEs ʹରͯ͠‫ޡ‬ೝࣝΛ؇࿨͢Δख๏ΛఏҊ͢Δɽຊ࣮‫Ͱݧ‬͸ɼσʔληοτͱͯ͠ 100 Ϋϥεͷࣗવը૾Λ‫ؚ‬Ή CIFAR-100 Λ༻͍ΔɽCIFAR-100 ͸ը૾αΠζ͕
32 × 32 ϐΫηϧͷ RGB ը૾Ͱɼ5 ສຕֶ͕श༻ɼ1 ສຕ͕ਪ࿦༻Ͱ͋ΔɽABN ͷϕʔεωοτϫʔΫͱͯ͠ 110
૚ͷ ResNet Λ࢖༻͢ΔɽϛχόοναΠζ͸ 128 ͱͯ͠
300 epoch ֶश͢Δɽઁಈͷ࡞੒ख๏͸ɼϵ = 8 ͷ FGSM
ͱ͢Δɽൺֱख๏͸௨ৗͷ ABN ͷਪ࿦݁Ռ‫ͼٴ‬ɼABN Λ
AEs Ͱ߈ܸͨ݁͠Ռͱ͢Δɽ

2. ؔ࿈‫ڀݚ‬

4.2. ࣮‫݁ݧ‬Ռ

ABN ͸ɼೖྗ͔Βಛ௃Λநग़͢Δ Feature extractor(FE)ɼ஫ࢹྖҬΛग़ྗ͢Δ Attention branch (AB)ɼ஫ࢹ
ྖҬΛ൓өͨ͠ಛ௃Ϛοϓ͔Β࠷ऴతͳग़ྗΛ͢Δ Perception branch (PB) Ͱߏ੒͞Ε͍ͯΔɽ஫ࢹྖҬΛֶश࣌ɼ‫ͼٴ‬ਪ࿦࣌ʹར༻͢Δ͜ͱͰɼߴਫ਼౓ͳೝ͕ࣝՄೳͰ
͋Δɽ·ͨɼ஫ࢹྖҬ͸ೖྗը૾ͷೝࣝ݁Ռʹର͢Δࢹ֮తཧղͷଅਐʹͭͳ͕Δɽ
Adversarial attack ͸ɼਓ͕஌֮ࠔ೉ͳઁಈΛը૾ʹ෇༩͢Δ͜ͱͰɼConvolutional neural network (CNN) ͷ‫ޡ‬ೝࣝΛ༠ൃ͢ΔɽFast Gradient Sign Method (FGSM) [2]͸ɼCNN ͷೝࣝ‫ ࠩޡ‬L(x, y, θ) Λೖྗը૾ x ͷ֤ըૉʹؔͯ͠ඍ෼͢Δ͜ͱͰޯ഑Λ‫ٻ‬ΊΔɽͦͯ͠ɼ‫ٻ‬Ί֤ͨըૉͷޯ഑Λ Sign ؔ਺ʹΑͬͯූ߸Λൈ͖ग़͢͜ͱͰɼઁ
ಈͷ୯ҐϕΫτϧΛ֫ಘ͢ΔɽAEs Ͱ͋Δ x̂ Λ࡞੒͢Δ
ʹ͸ɼ୯ҐϕΫτϧʹઁಈͷ‫౓ڧ‬Λද͢ ϵ ∈ [0, 255] Λ৐ࢉͯ͠ x ʹՃࢉ͢ΔɽઁಈΛՃࢉ͢Δ͜ͱͰɼ‫ࠩޡ‬Λ࠷େ
Խ͢Δ AEs Λ࡞੒͢Δ͜ͱ͕ՄೳͱͳΔɽFGSM ͷҰ࿈ͷྲྀΕ͸ɼࣜ (1) ʹΑͬͯද‫͢ݱ‬Δ͜ͱ͕Ͱ͖Δɽ

3. ఏҊख๏

x̂ := x + ϵ · sign (∇x L (x, y, θ))ද 1 ʹࣔ͢Α͏ʹɼABN ͸ AEs ͷӨ‫ʹڹ‬ΑΓೝࣝੑೳ͕ஶ͘͠௿Լͨ͠ɽҰํɼఏҊख๏͸ɼಗ໊Խͨ͠ AB͕ग़ྗͨ͠஫ࢹྖҬΛಛ௃Ϛοϓʹ൓ө͢Δ͜ͱͰɼ60 ϙΠϯτҎ্ͷੑೳ޲্Λ֬ೝͨ͠ɽAEs Λ༻͍ͳ͍৔߹
(ABN) ͷਪ࿦݁Ռͱൺֱ͢ΔͱɼఏҊख๏͸ 11 ϙΠϯτఔ౓ͷྼԽʹཹΊΔ͜ͱΛՄೳͱͨ͠ɽ
ද 1 : ೝࣝਫ਼౓ͷൺֱ (%)
ABN ABN+AEs
Ours

top-1
top-5

65.22
87.49

(1)

(a) Original images(b) ABN：Backward(c) Ours
Perception
branchℒ!"#!
Weight parameters )Perturbation δAdversarial examples "'Featureextractor

Perception
branchਤ 2 : ஫ࢹྖҬͷྫ

Label

Featureextractor

Input image "

+$%

1.82
9.12

஫ࢹྖҬΛਤ 2 ʹࣔ͢ɽਤ 2 ΑΓɼఏҊख๏͸ AEs Λೖྗ͍ͯ͠Δʹ΋ؔΘΒͣɼൃՐ͢ΔྖҬ͸ ABN ͱ΄΅ಉ͡Ͱ͋Δ͜ͱ͕֬ೝͰ͖Δɽ͜ͷ݁Ռ͔ΒɼAB ͸ AEs
ʹΑͬͯ֎෦͔Β߈ܸ͞Εͯ΋ɼೖྗը૾ʹରͯ͠ਖ਼͍͠
ྖҬΛ஫ࢹ͢Δ͜ͱ͕Ͱ͖Δɽ·ͨɼAEs ʹΑΓ FE ͕มಈͨ͠ಛ௃ྔΛ‫ͯ͠ࢉܭ‬΋ɼਖ਼͍͠஫ࢹྖҬΛ‫ڧ‬ௐ͢Δ͜
ͱͰɼੑೳΛҡ࣋Ͱ͖Δ͜ͱ͕Θ͔ͬͨɽ

ABN ͸ Adversarial attack Λ૝ఆͨ͠ઃ‫͍ͨͳͰܭ‬Ίɼ
AEs Λೖྗ͢Δͱஶ͘͠ੑೳ͕ྼԽ͢Δɽͦ͜Ͱຊ‫Ͱڀݚ‬͸ɼAB Λಗ໊Խ͢Δɽͦͯ͠ɼFE ͱ PB Λ߈ܸର৅ͱ
ͯ͠ AEs Λ࡞੒͢ΔɽAEs ͷਪ࿦࣌͸ AB Λ௥Ճͯ͠࠷
ऴ݁ՌΛ‫ٻ‬ΊΔɽఏҊख๏͸ҎԼʹࣔ͢ 3 εςοϓ͔Β੒
Δɽ
Step1 ABN Λ௨ৗͷֶशσʔλΛ༻͍ͯ‫܇‬࿅ͯ͠ɼ༏ल
ʹͳͬͨ AB Λ੾Γ཭ͯ͠ಗ໊Խ͢Δɽ
Step2 PB ͷग़ྗͱ‫ࢣڭ‬৴߸ͱͷ‫ࠩޡ‬Λೖྗը૾ʹؔͯ͠
ඍ෼͢Δ͜ͱͰ‫ࠩޡ‬Λ࠷େʹ͢Δ୯ҐϕΫτϧΛ‫ٻ‬ΊΔɽ
Step3 ಗ໊Խͨ͠ AB Λ෇͚଍ͨ͠ωοτϫʔΫʹɼAEs
Λೖྗ͠ɼ஫ࢹྖҬΛ‫ٻ‬ΊΔɽͦͯ͠ɼ஫ҙྖҬΛॏΈ෇ͨ͠ಛ௃ϚοϓΛ PB ʹೖྗͯ͠ਪ࿦Λߦ͏ɽ
ਤ 1 ʹ Step2 ͱ Step3 ͷҰ࿈ͷྲྀΕΛࣔ͢ɽ
：Forward

77.18
93.54

( "'

Weight parameters )Anonymized
Attention
branchਤ 1 : ఏҊख๏ͷྲྀΕ

4. ධՁ࣮‫ݧ‬ఏҊख๏ͷ༗ޮੑΛࣔͨ͢ΊʹɼఆྔతͳೝࣝੑೳͷධՁ‫ͼٴ‬ఆੑతͳ஫ࢹྖҬͷධՁΛ͢Δɽ

5. ͓ΘΓʹ
ຊ‫Ͱڀݚ‬͸ɼਓ޻తͳઁಈͰ͋Δ Adversarial examples(AEs) ʹ‫ ͳ݈ؤ‬Attention branch network(ABN) ΛఏҊͨ͠ɽ࣮‫ݧ‬ΑΓɼAEs ͕ೖྗ͞Εͯ΋஫ࢹྖҬͷมಈ͕ඍখͰ͋Δ͜ͱΛ֬ೝͨ͠ɽ·ͨɼೝࣝ݁Ռ͸ೖྗը૾ʹରͯ͠ਖ਼֬ͳྖҬΛ‫ڧ‬ௐ͢Δ͜ͱͰɼ͋Δఔ౓ҡ࣋Ͱ͖
Δ͜ͱ͕൑໌ͨ͠ɽࠓ‫ޙ‬͸ɼຊఏҊख๏ͷߋͳΔ෼ੳΛ͢Δ༧ఆͰ͋Δɽࢀߟจ‫ݙ‬[1] H. Fukui, et al., “Attention Branch Network: Learning of Attention Mechanism for Visual Explanation”, CVPR, 2019.
[2] l. Goodfellow, et al., “Explaining and Harnessing
Adversarial Examples”, ICLRɼ2014.

஌ࣝৠཹʹ͓͚Δ Multiple Teacher Network ͷબ୒ࢦಋ‫ڭ‬तɿ౻٢߂࿱

ER17048 தౡ྅༎1. ͸͡Ίʹ
஌ࣝৠཹ͸ Teacher Network ͷ஌ࣝΛ Student Networkʹ఻͑Δ͜ͱͰɼਫ਼౓௿ԼΛ཈੍ͭͭ͠ϞσϧαΠζΛѹॖ͢Δख๏Ͱ͋Δ. Teacher Network ͱ Student Network ͷϞσϧαΠζʹେ͖ͳΪϟοϓ͕͋ΔͱɼStudent Networkͷਫ਼౓͕޲্͠ͳ͍໰୊͕͋Δɽຊ‫Ͱڀݚ‬͸ɼαΠζͷҟ
ͳΔෳ਺ͷ Teacher Network Λ༻͍Δ Multiple Teacher
Network ΛఏҊ͠ɼ஌ࣝৠཹͷߴਫ਼౓ԽΛ໨తͱ͢Δɽ

2. ஌ࣝৠཹͱ໰୊఺
஌ࣝৠཹʹ͸ɼTeacher Network ͷग़ྗΛར༻͢Δ Knowledge Distillation (KD)[1] ͱɼதؒ৘ใΛར༻͢Δ Contrastive Representation Distillation(CRD)[2] ͕͋Δɽද
1 ʹ Student Network Λ ResNet20 ʹͨ͠৔߹ͷ KD ʹΑ
Δৠཹ‫ޙ‬ͷਫ਼౓ͱ Teacher Network ͷϞσϧαΠζ (ύϥϝʔλ਺) Λࣔ͢ɽTeacher Network ͕ WRN40 2 ΑΓେ͖ͳϞσϧαΠζʹͳΔͱɼେ෯ͳਫ਼౓௿ԼΛট͘͜ͱ͕Θ
͔Δɽຊ‫Ͱڀݚ‬͸ɼ͜ͷ໰୊͕ൃੜ͢Δ Teacher Network
Λ Large Teacher Network(LTN)ɼൃੜ͠ͳ͍ Teacher
Network Λ Medium Teacher Network(MTN) ͱ‫ݺ‬শ͢
Δɽ͜ͷ໰୊Λղܾ͢Δख๏ͱͯ͠ɼTeacher Network ͱStudent Network ؒʹ Teacher Assistant(TA) ͱ‫Ϳݺ‬ϞσϧΛ഑ஔ͠ɼஈ֊తʹৠཹ͢Δख๏ [3] ͕ఏҊ͞Ε͍ͯ
Δ͕ɼஈ֊తͳֶशΛඞཁͱ͢Δɽ

·ͨɼ࠷ऴతͳଛࣦ͸ࣜ (3) ʹࣔ͢௨ΓɼશͯͷଛࣦΛ૯࿨ͨ͠஋ʹͳΔɽ
n#
Ltotal =LiKDorCRD(3)
i=0

4. ධՁ࣮‫ݧ‬ఏҊख๏ͷ༗ޮੑΛධՁ͢ΔͨΊɼධՁ࣮‫ݧ‬Λߦ͏ɽTAͱ΋ൺֱ͠ධՁ͢Δɽ

4.1. ࣮‫֓ݧ‬ཁσʔληοτʹ͸ CIFAR-100 Λ࢖༻͠ɼֶश࣌ͷΤϙοΫ਺͸ 200ɼόοναΠζ͸ 64 ͱ͢ΔɽֶशϞσϧ͸ Student Network ʹ ResNet20ɼLTN ʹ WRN40 2ʙ
WRN28 10ɼMTN ʹ WRN28 1ʙWRN28 2 Λ༻͍Δɽ

4.2. ࣮‫݁ݧ‬Ռ
ද 2 ͸ 2 ͭͷ Teacher Network Λ༻͍ͯৠཹͨ͠ਫ਼౓
Λࣔ͢ɽTN1ɼ2ɼ3 ʹ͸ͦΕͧΕҟͳΔϞσϧΛ࢖༻͠ɼTN3 ͸ TN2 ΍ TN1 ΑΓେ͖͘ɼTN2 ͸ TN1 ΑΓେ͖͍ϞσϧͰ͋ΔɽTeacher Network ͷ૊Έ߹Θͤ͸ɼMTN
ͷΈɼMTN+LTNɼLTN ͷΈͷ 3 छྨ͋Δɽද 2: 2 ͭͷ Teacher Network Ͱৠཹͨ͠ਫ਼౓
ఏҊख๏

Teacher Network
TN1

TN2

ResNet32

WRN28 271.12 71.26 72.13

Teacher Network WRN28 1 ResNet32 WRN28 2 WRN40 2 WRN28 6 WRN28 10

MTN+LTN ResNet32

WRN28 670.83 71.24 71.73

ύϥϝʔλ਺

371,347

466,457 1,147,251 2,248,991 13,155,603 36,497,171

MTN+LTN ResNet32 WRN28 10 70.24 71.37 71.61

ਫ਼౓

71.27

71.12

LTN ͷΈ

ද 1: ResNet20 ʹର͢Δৠཹ‫ޙ‬ਫ਼౓

70.98

70.25

MTN ͷΈ

70.14

69.60

3. ఏҊख๏
ຊ‫Ͱڀݚ‬͸ɼਤ 1 ʹࣔ͢Α͏ʹෳ਺ͷ Teacher Network͔Βৠཹ͢Δ Multiple Teacher Network ΛఏҊ͢Δɽ͜
ͷͱ͖ɼͲͷΑ͏ʹ Teacher Network Λબ୒͢ΔͱޮՌతͰ͋Δ͔ௐࠪ͢ΔɽఏҊख๏Ͱ͸ɼTeacher Network ͷ૊Έ߹ΘͤΛ MTN+LTN ͱ͠ɼMTN ʹ Student Network
ͱ LTN ͷ‫͠౉ڮ‬తͳ໾ׂΛ‫ظ‬଴͢Δɽ·ͨɼMTN ͷΈͰৠཹΛߦͬͨ৔߹ΑΓ΋ɼLTN Λ‫ؚ‬Ίͯৠཹ͢Δ͜ͱͰɼͦͷ஌ࣝ΋‫͖Ͱ༻׆‬Δͱߟ͑Δɽ
ResNet32

WRN28_6

ResNet32

WRN28_6

WRN28_10

M

L

M

L

L

ਤ 1: ఏҊख๏ͷ֓೦ਤຊ‫Ͱڀݚ‬ͷ Student Network ͱ Teacher Network ؒͷৠཹख๏ʹ͸ɼKDɼCRD ͷ͍ͣΕ͔Λ༻͍ΔɽKD ͷଛࣦؔ਺Λࣜ (1) ʹࣔ͢ɽ

LKD = αLhard + (1 − α) τ 2 Lsof t(1)͜͜Ͱɼα ͸܎਺ɼτ ͸Թ౓ύϥϝʔλΛࣔ͢ɽCRD ͷଛࣦؔ਺Λࣜ (2) ʹࣔ͢ɽTS

eg g /τ
]T g S /τ
N
g
e
+M
!
"
T S
eg g /τ
+N Eq(gT ,gS |C=0) [1 − log]NegT gS /τ + M
LCRD = Eq(gT ,gS |C=1) [log(2)͜͜Ͱɼτ ͸Թ౓ύϥϝʔλɼM ͸σʔληοτͷΧʔσΟφϦςΟͰɼσʔλͷछྨ͕Ͳͷ͘Β͍͋Δ͔ͷ౓߹͍Λද͍ͯ͠ΔɽN ͸ೖྗσʔλͷඇྨࣅϖΞͷ਺Ͱ͋Δɽg S
ͱ g T ͸֤ωοτϫʔΫͷग़ྗ૚खલ͔Βநग़ͨ͠ग़ྗΛઢ‫ܗ‬ม‫ͤ͞׵‬ɼL2 ϊϧϜͰਖ਼‫ن‬Խͨ͠΋ͷͰ͋Δɽ

KD

CRD

WRN28 6 WRN28 10 69.37 69.92 70.75

ද 2 ͔ΒɼMTN ͷΈͷ৔߹͕࠷΋ਫ਼౓͕ߴ͘ɼLTN ͷΈͷ৔߹͕࠷΋௿͘ͳͬͨɽMTN+LTN ͷ৔߹͸ɼLTN
ͷΈΑΓਫ਼౓͕ߴ͍ɽҎ্ΑΓɼ2 ͭͷ Teacher Networkͷ͏ͪ 1 ͭΛ MTN ʹ͢Δ͜ͱͰɼStudent Network ͱ
LTN ؒͷৠཹͰੜ͡Δ໰୊Λճආ͢Δ͜ͱ͕Ͱ͖Δɽද 3: 3 ͭͷ Teacher Network Ͱৠཹͨ͠ਫ਼౓
ఏҊख๏

Teacher Network
TN1

TN2

TN3

TA

KD

CRD

MTN ͷΈ WRN28 1 ResNet32 WRN28 2 71.18 71.92 72.19
MTN+LTN ResNet32 WRN28 2 WRN28 6 70.24 71.06 71.83
MTN+LTN WRN28 2 WRN28 6 WRN28 10 68.87 70.15 71.00
LTN ͷΈ

: KD or CRD

TA

WRN40 2 WRN28 6 WRN28 10 68.99 70.01 70.55

ද 3 ͸ 3 ͭͷ Teacher Network Λ༻͍ͯৠཹͨ͠ਫ਼౓
Λࣔ͢ɽද 3 ͔ΒɼMTN+LTN ͷ৔߹ɼਫ਼౓௿ԼΛ཈੍
͢Δ͜ͱ͕Ͱ͖ͳ͍͜ͱ͕͜ͱ͕Θ͔ΔɽMTN ͷΈͷ৔߹͕࠷΋ߴਫ਼౓Ͱ͋ΔɽҎ্ΑΓɼTeacher Network ਺Λ૿΍ͯ͠΋ LTN ͕͋Δ৔߹ɼߴਫ਼౓Խ͕೉͍͜͠ͱ͕൑
໌ͨ͠ɽ

5. ͓ΘΓʹ
ຊ‫Ͱڀݚ‬͸ɼೳྗࠩͷ͋Δ Student Network ͱ Teacher
Network ؒͷৠཹͷࡍʹɼMTN+LTN ͷ૊߹ͤΛ‫ؚ‬ΊͯͦΕΒΛಉ࣌ʹৠཹͤ͞Δ͜ͱΛఏҊͨ͠ɽ͜ΕʹΑΓɼ
Teacher Network Λ 2 ͭ࢖༻͢Δ৔߹͸ਫ਼౓ͷ௿ԼΛ཈
͑ɼMTN ͷΈͷਫ਼౓ʹඖఢ͠ɼ͞Βʹ Teacher Network͕ 1 ͭͷΈͰৠཹΛߦͬͨ৔߹ΑΓ΋ਫ਼౓͕޲্͢Δ͜ͱ
Λ֬ೝͨ͠ɽ·ͨɼTA ͱҟͳΓஈ֊తͳֶशΛඞཁͱͤͣɼͦͷ෼ֶश࣌ؒΛ୹ॖͰ͖Δ͜ͱ΋֬ೝͨ͠ɽ

ࢀߟจ‫ݙ‬[1] G. Hinton, et al., “Distilling the knowledge in a neural network”, NeurIPS Workshop, 2015.
[2] YɽTian, et al., “Contrastive Representation Distillation”, ICLR, 2020.
[3] S. Mirzadeh, et al., “Improved Knowledge Distillation via Teacher Assistant”, AAAI, 2020.

Refined Consistency ʹΑΔ஌ࣝৠཹΛ༻͍ͨ൒‫͋ࢣڭ‬Γֶशࢦಋ‫ڭ‬तɿ౻٢߂࿱

ER17074 ଜຊՂོ1. ͸͡Ίʹ

jFinal loss൒‫͋ࢣڭ‬Γֶश͸ɼϥϕϧ͋Γσʔλͱϥϕϧͳ͠σʔλͷ૒ํΛ༻ֶ͍ͨश๏Ͱ͋Δɽ൒‫͋ࢣڭ‬Γֶशख๏ͱ
ͯ͠ɼ2 ͭͷωοτϫʔΫؒͰ஌ࣝΛసҠ͢Δ Dual Student(DS) [1]ɼDS ͷωοτϫʔΫ਺Λ 4 ͭҎ্ʹ֦ுͨ͠
Multiple Student(MS) [1] ͕ఏҊ͞Ε͍ͯΔɽMS ͸ DSʹൺ΂ͯߴ͍ਫ਼౓Ͱ͋Δ͕ɼશͯͷωοτϫʔΫؒͰͷ஌ࣝసҠΛҰ౓ʹߦ͏͜ͱ͕Ͱ͖ͳ͍ͨΊɼֶशޮ཰͕ѱ͍ɽͦ͜Ͱຊ‫Ͱڀݚ‬͸ɼޮ཰తͳ஌ࣝసҠํ๏ʹΑΔਫ਼౓޲্Λ໨తͱͯ͠ɼ1 ౓ʹશͯͷωοτϫʔΫؒͰ஌ࣝΛసҠ
͢Δ Refined Consistency ΛఏҊ͢Δɽ·ͨɼҰ؏ੑʹ‫ج‬͍ͮͨ൒‫͋ࢣڭ‬Γֶशख๏ʹར༻͞ΕΔσʔλ֦ுख๏Λద༻͠ɼਫ਼౓มԽΛධՁ͢Δɽ

2.Dual Student
DS ͸ɼ2 ͭͷωοτϫʔΫΛ༻͍ͨ൒‫͋ࢣڭ‬Γֶशख
๏Ͱ͋Δɽ֤ωοτϫʔΫʹ͸ɼಉ͡ೖྗʹରͯ͠ҟͳΔ
ઁಈΛՃ͑ͨ 2 ͭͷೖྗΛ༩͑Δɽͦͯ͠ɼઁಈʹରͯ͠‫ͳʹ݈ؤ‬ΔΑ͏ֶश͢Δɽ͞Βʹɼ҆ఆੑͱ͍͏ࢦඪΛ༻͍ͯωοτϫʔΫग़ྗͷҰ؏ੑΛධՁ͠ɼ҆ఆੑ͕ߴ͍
ωοτϫʔΫ͔Β༏Εͨ஌ࣝΛసҠ͍ͤͯ͞Δɽ஌ࣝ͸ଟ͘ͷωοτϫʔΫ͔ΒసҠͤ͞Δํ͕ߴ͍ਫ਼౓Λୡ੒Ͱ͖
ΔɽMS ͸ DS Λ֦ு͠ɼ4 ͭͷωοτϫʔΫΛ༻͍͍ͯ
ΔɽMS ͸ɼ1 ౓ʹϥϯμϜͳ 2 ͭͷωοτϫʔΫΛ༻͍
ͯɼDS ͱಉ༷ͷํ๏Ͱֶश͢Δɽ

3. ఏҊख๏
ຊ‫Ͱڀݚ‬͸ɼ1 ౓ʹશͯͷωοτϫʔΫؒͰ஌ࣝΛసҠ
͢Δ Refined Consistency ΛఏҊ͢ΔɽMS ͱఏҊख๏ʹ͍ͭͯɼ1 ΠλϨʔγϣϯ͝ͱͷ஌ࣝసҠͷํ޲Λਤ 1 ʹࣔ͢ɽఏҊख๏ͷֶशํ๏Λਤ 2 ʹࣔ͢ɽ
Network 1

n+1
iteration

Network 3

Knowledge
Translationniteration

Network 2

Network 1niteration

Network 3(a) MS

Network 2

Network 4(b) Refined Consistencyਤ 1: ஌ࣝసҠํ޲ͷྫ (ωοτϫʔΫ਺ 4)
ೖྗ x ʹର͢ΔωοτϫʔΫ i ͷଛࣦؔ਺ Li (x) Λࣜ(1) ʹࣔ͢ɽ

Li (x) = Licls + λ1 Licon + λ2 Lista(1)͜͜Ͱɼλ1 , λ2 ͸ॏΈ܎਺Ͱ͋ΓɼLcls ͸ਪଌ஋ͱϥϕϧؒͷަࠩΤϯτϩϐʔͰ͋Γɼx ͕ϥϕϧ͋ΓσʔλͰ͋Δ৔߹ͷΈ‫͢ࢉܭ‬ΔɽLcon ͸Ұ؏ੑଛࣦɼLsta ͸҆ఆԽଛࣦΛද͢ɽ҆ఆԽଛࣦ͸ɼ֤ωοτϫʔΫͷҰ؏ੑଛࣦΛ༻͍ͯ‫͢ࢉܭ‬Δɽ
Ұ؏ੑଛࣦ͸ɼҟͳΔઁಈΛೖྗʹ༩͑ͨͱ͖ͷࣄ‫֬ޙ‬཰ؒͷ‫͋Ͱࠩޡ‬Γɼ஋͕খ͍͞΄ͲωοτϫʔΫ͕ઁಈʹ‫͋Ͱ݈ؤ‬Δ͜ͱΛද͢ɽҰ؏ੑଛࣦΛࣜ (2) ʹࣔ͢ɽ

Licon (x) = Ex∈D ||f (θi , x + ζ1 ) − f (θi , x + ζ2 )||2

LabeledNetwork

Network

Consistency

Unlabeled

Network

Network

ਤ 2: Refined Consistency ͷֶशํ๏(ωοτϫʔΫ਺ 4)
f (θi , x + ζ1 ) ͱ f (θi , x + ζ2 ) ͷΫϥε͕Ұகͨ͠৔߹ʹਅͱͳΔɽ

4. ධՁ࣮‫ݧ‬
4.1. ࣮‫֓ݧ‬ཁఏҊख๏ͱ MS ͷਫ਼౓Λൺֱ͢Δɽσʔληοτʹ CIFAR100ɼϕʔεϞσϧʹ 13 ૚ͷ CNN Λ࢖༻͢Δɽֶशճ਺
͸ 350 epochɼωοτϫʔΫ਺͸ 4ɼ8 ͱ͢Δɽσʔλ֦ுͱͯ͠ɼࠨӈ൓సɼϥϯμϜͳըૉ਺ฏߦҠಈͨ͠ͷͪʹ൓ࣹύσΟϯάΛߦ͏ɽCIFAR-100 ͸ɼΫϥε਺͕ 100Ͱ͋Γɼֶश༻ʹ 50,000 ຕɼਪ࿦༻ʹ 10,000 ຕͷը૾Ͱߏ੒͞ΕΔɽຊ࣮‫Ͱݧ‬͸ɼֶश༻ը૾ͷ͏ͪ 10,000 ຕΛϥϕϧ͋Γσʔλɼଞ 40,000 ຕΛϥϕϧͳ͠σʔλͱ͢ΔɽҰ؏ੑʹ‫͍ͨͮج‬൒‫͋ࢣڭ‬Γֶशख๏Ͱ͸ɼRandAugment(RA) [2] ͳͲͷେ͖ͳઁಈΛ෇༩͢Δσʔλ֦ுख๏Λѻ͏͜ͱͰਫ਼౓͕޲্͢Δ [3]ɽͦ͜ͰɼRA ͷ༗ແʹ
ΑΔਫ਼౓มԽΛධՁ͢Δɽ

4.2. ࣮‫݁ݧ‬Ռ

Knowledge
Translation
Network 4

Dataset(2)

͜͜ͰɼD ͸σʔληοτɼζ1 , ζ2 ͸ϥϯμϜੑͷ͋Δઁಈɼf (θ, x) ͸ॏΈ θ Λ΋ͭωοτϫʔΫͷࣄ‫཰֬ޙ‬Λද͢ɽ҆ఆԽଛࣦ͸ɼҰ؏ੑଛࣦΛ஌ࣝͷࢦඪͱͯ͠ɼ࠷΋Ұ؏ੑଛࣦ͕খ͍͞ωοτϫʔΫ θs ͔ΒଞͷશͯͷωοτϫʔΫʹ஌ࣝΛసҠ͢ΔɽసҠઌͷωοτϫʔΫΛ θi ͱ
͢Δͱɼ஌ࣝ͸ɼࣄ‫཰֬ޙ‬Λ͚ۙͮΔΑ͏ʹֶश͢Δ͜ͱͰసҠͤ͞Δɽ҆ఆԽଛࣦΛࣜ (3) ʹࣔ͢ɽ!
0
(A = ∅)i
Lsta (x) =
(3)
i
s
2
||f (θ , x) − f (θ , x)|| (otherwise)
͜͜ͰɼA = {i | R(f (θi , x + ζ1 ), f (θi , x + ζ2 )) = True}Ͱ͋ΓɼRix ͸ೖྗ x ʹର͠ɼઁಈΛ෇༩ͨ͠ࡍͷग़ྗఏҊख๏͓Αͼ MS ͷਫ਼౓ɼ֤ख๏ʹରͯ͠ RA Λద༻ͨ͠ਫ਼౓Λද 1 ʹࣔ͢ɽ֤݁Ռ͸ 5 ճࢼߦͨ࣌͠ͷਖ਼ղ཰ͷฏ‫ۉ‬஋ͱඪ४ภࠩͰ͋Δɽද 1 ͔ΒɼఏҊख๏͸ MS
ͱൺ΂ͯɼωοτϫʔΫ਺ 4 Ͱ 0.26 ptɼωοτϫʔΫ਺
8 Ͱ 0.31 pt ਫ਼౓͕ߴ͍ɽΑͬͯɼఏҊख๏ʹ͓͚Δ஌ࣝసҠ͸༗ޮͰ͋Δͱ‫͑ݴ‬Δɽ͞ΒʹɼRA ͱఏҊख๏Λ૊Έ߹ΘͤΔ͜ͱͰɼͦͷޮՌ͸࠷େͱͳΓ 68.18%·Ͱ޲্ͨ͠ɽද 1: CIFAR-100 Λ༻͍ͨਫ਼౓ൺֱ [%]
ख๏

MS
ఏҊख๏

֦ு

RA
RA

ωοτϫʔΫ਺
4
8

66.70±0.15
66.43±0.14
66.96±0.12
66.96±0.04

66.93±0.09
66.34±0.10
67.24±0.06
68.18±0.10

5. ͓ΘΓʹ
ຊ‫Ͱڀݚ‬͸ɼ1 ౓ʹશͯͷωοτϫʔΫؒͰ஌ࣝΛసҠ
͢Δ Refined Consistency ΛఏҊͨ͠ɽධՁ࣮‫Ͱݧ‬͸ɼఏҊख๏͕ MS ʹൺ΂ͯߴਫ਼౓Ͱ͋Δ͜ͱΛࣔͨ͠ɽࠓ‫ޙ‬͸ɼ஌ࣝసҠάϥϑ΁ͷԠ༻Λߦ͏༧ఆͰ͋Δɽࢀߟจ‫ݙ‬[1] K. Zhanghan, et al., “Dual Student: Breaking theLimits of the Teacher in Semi-supervised Learning”,
ICCV, 2019.
[2] E. Cubuk, et al., “RandAugment: Practical automated data augmentation with a reduced search
space”, NeurIPS, 2019.
[3] K. Sohn, et al., “FixMatch: Simplifying SemiSupervised Learning with Consistency and Confidence”, NeurIPS, 2020.

ਂ૚‫ڧ‬Խֶशʹ͓͚ΔՁ஋ɾํࡦϕʔεͷ༗ޮੑͷ܏޲ௐࠪࢦಋ‫ڭ‬तɿ౻٢߂࿱

ER17073 ଜ্ɹང1. ͸͡Ίʹ

3.1. ௐࠪ‫ڥ؀‬ਂ૚‫ڧ‬Խֶश͸ɼਂ૚ֶशͱ‫ڧ‬ԽֶशΛ૊Έ߹Θֶͤͨशํ๏Ͱ͋Δɽ‫ڧ‬Խֶश͸ɼ͋Δ‫͔ڥ؀‬Β֫ಘͰ͖Δใुͷ࠷େԽΛ໨తͱ͠ɼ࠷దͳߦಈΛ֫ಘ͢Δֶशํ๏Ͱ͋
Δɽਂ૚‫ڧ‬Խֶशख๏ʹ͸ɼେผͯ͠Ձ஋ϕʔεͱํࡦϕʔε͕͋Δɽਂ૚‫ڧ‬Խֶश͸ਓͷੑೳΛ௒͑Δ͜ͱ΋͋Δ͕ɼ
ख๏͝ͱʹֶश‫ڥ؀‬΍ֶश৚͕݅ҟͳΔͨΊɼ֤ख๏ʹͲͷΑ͏ͳ܏޲͕͋Δ͔໌֬Ͱͳ͍ɽͦ͜Ͱຊ‫Ͱڀݚ‬͸ɼՁ஋΂ʔεɾํࡦϕʔεͰ୅දతͳΞϧΰϦζϜͷ܏޲ͷௐࠪΛߦ͏ɽ‫ʹڥ؀‬͸ɼOpen AI ͕ఏ‫͢ڙ‬Δ Atari 2600 ͷήʔϜλεΫͷ͏ͪ BreakoutɼMs.PacmanɼPong, Qbert, Spaceinvaders Λ࢖༻͢Δ. ͜ΕΒͷ‫ڥ؀‬͸ใुͷछྨ΍ɼ೉қ౓͕ҟͳΔΑ͏ʹબ୒ͨ͠ɽBreakout ͸ɼύυϧͰϘʔϧΛଧͪฦ͠ϒϩοΫΛ่͢ήʔϜͰ͋ΔɽMs.Pacman ͸ɼΰʔετΛճආ͠ͳ͕ΒΫοΩʔΛ৯΂ΔήʔϜͰ͋ΔɽPong
͸ɼCPU ͱϗοέʔΛߦ͏ήʔϜͰ͋ΔɽQbert ͸ɼδϟϯϓͯ͠ϒϩοΫͷ৭Λมߋ͢ΔήʔϜͰ͋ΔɽSpaceinvades ͸ɼΠϯϕʔμΛଧͪൈ͘ήʔϜͰ͋Δɽ֤ख๏͸
1.0 ʷ 107 ΤϐιʔυͷֶशΛߦ͍ɼ1000 ճςετͨ࣌͠ͷฏ‫ۉ‬είΞΛࢉग़͢Δɽ·ͨɼϥϯμϜʹߦಈΛબ୒ͨ͠৔߹ͷείΞͱ΋ൺֱ͢Δɽ

2. ਂ૚‫ڧ‬Խֶशਂ૚‫ڧ‬Խֶश͸ɼਂ૚ֶशͱ‫ڧ‬ԽֶशΛ૊Έ߹Θֶͤͨशํ๏Ͱ͋ΔɽҎԼʹɼՁ஋ϕʔεͱํࡦϕʔεͷख๏ʹड़΂Δɽ

2.1. Ձ஋ϕʔεՁ஋΂ʔε͸ɼߦಈͷՁ஋ͷ࠷େԽ͢Δ͜ͱΛ໨తͱͯ͠ɼQ ֶशΛ༻ֶ͍ͯश͢ΔɽQ ֶश͸ TD ‫ʹࠩޡ‬ΑΓɼߦಈͷՁ஋Λߋ৽͢ΔɽTD ‫ ࠩޡ‬δ ͸͋Δঢ়ଶ s Ͱߦಈ aΛߦ͏ͱ͖ͷߦಈͷՁ஋ V (s) ͱɼߦಈ‫ޙ‬ͷঢ়ଶ s′ ʹ͓͚ΔߦಈͷՁ஋ V (s′ ) ͷࠩΛ΋ͱʹࣜ (1) ͷΑ͏ʹࢉग़͢Δɽ
TD ‫͕ࠩޡ‬େ͖͍΄Ͳঢ়ଶ s ͰͷߦಈͷՁ஋͕ߴ͘ͳΔɽ

δ = r + γV (s′ ) − V (s)(1)͜͜Ͱɼࣜ (1) ͷɼr ͸ใुɼγ ͸ɼׂҾ཰Ͱ͋Δɽ
Deep Q-Network(DQN)[1]ɿߦಈͷՁ஋ (Q ஋) Λ৞ΈࠐΈχϡʔϥϧωοτϫʔΫʹΑΔۙࣅؔ਺Ͱද‫ͨ͠ݱ‬‫ڧ‬Խֶशख๏Ͱ͋Δɽ·ͨɼExperience Replay ΍ Target
Q-NetworkɼReward Clipping Λಋೖͯ͠ɼ๲େͳঢ়ଶ਺
ͷը૾ೖྗʹରԠ͍ͯ͠Δɽ

Categolical DQN[2]ɿDQN Λ‫͠ͱج‬ɼߦಈՁ஋Λ෼෍ͱͯ͠ද‫͢ݱ‬Δ‫ڧ‬Խֶशख๏Ͱ͋Δɽ

2.2. ํࡦϕʔε3.2. ௐࠪ݁Ռ
ද 1 ʹ֤ήʔϜʹ͓͚Δ֤ख๏ͷฏ‫ۉ‬είΞΛࣔ͢ɽද 1
͔ΒɼBreakoutɾQbertɾSpaceinvaders Ͱ͸ɼCategorical
DQNɼPPO ͕ߴείΞΛ֫ಘͨ͠ɽCategolical DQN ͸ߦಈՁ஋Λ෼෍Ͱද‫͢ݱ‬Δ͜ͱʹΑΓɼֶशʹ‫·ؚ‬Εͳ͍ঢ়ଶʹର͢ΔߦಈՁ஋Λਖ਼͘͠‫ٻ‬ΊΔ͜ͱ͕Ͱ͖ΔͨΊɼείΞ͕޲্ͨ͠ͱߟ͑ΒΕΔɽPPO ͸ํࡦͷมԽྔͷΫϦοϐϯάޮՌʹΑΓֶशʹ‫·ؚ‬ΕΔঢ়ଶʹΦʔόʔϑΟοτ͠ͳ͍ͨΊɼείΞ͕޲্ͨ͠ͱߟ͑ΒΕΔɽ͜ͷ̏ͭͷήʔϜͰ͸ɼՁ஋ϕʔεɾํࡦϕʔεͷͲͪΒ΋༗ޮͰ
͋Δɽ
Ms.Pacman Ͱ͸Ձ஋ϕʔεͷख๏ͷํ͕ߴ͍είΞΛ֫ಘͨ͠ɽ͜Ε͸ɼMs.Pacman ͕Τϐιʔυ‫ޙ‬൒Ͱ֫ಘ
Ͱ͖Δใु͕গͳ͘ͳΔͱ࣌ɼήʔϜͷਐߦʹ߹Θͤͯ֫
ಘͰ͖Δใु͕‫ݮ‬ΔͨΊɼํࡦϕʔεͰ͸ద੾ͳߦಈΛಘ
Δ͜ͱ͕Ͱ͖ͳ͍ɽPong ͸ɼશͯͷख๏ʹ͓͍ͯߴείΞΛ֫ಘͰ͖͍ͯΔɽ͜Ε͸ɼ‫ڥ؀‬ͷλεΫ͕༰қͰ͋Δ
ͨΊɼख๏ʹΑΔେ͖ͳ͕ࠩग़ͳ͔ͬͨͨΊͰ͋Δͱߟ͑ΒΕΔɽ
ද 1 : 1000 Τϐιʔυؒͷฏ‫ۉ‬είΞ

Ձ஋ϕʔε

ํࡦϕʔε͸ɼߦಈͷ‫ن‬ൣͱͳΔํࡦΛ࠷దԽ͢Δ͜ͱ
Λ໨తͱͯ͠ɼํࡦޯ഑๏Λ༻ֶ͍ͯश͢Δɽํࡦޯ഑๏
͸ํࡦ πθ ͷύϥϝʔλ θ Ͱɼऩӹ (ใुͷ૯࿨) ͷ‫ظ‬଴஋Λඍ෼ͯ͠ޯ഑Λࢉग़ࣜ͠ (2) ͷΑ͏ʹߋ৽͢Δɽࣜ (2)ͷ α ͸ֶश཰ΛɼJ ͸‫ظ‬଴ऩӹΛࣔ͢ɽθt+1 ← θt + α∇θ J(θ)(2)

Asynchronous Advantage Actor-Critic(A3C)[3]ɿ
ෳ਺ͷ worker Λ༻͍ͯඇಉ‫ظ‬తʹύϥϝʔλߋ৽Λߦ͏
Asynchronousɼ2 εςοϓҎ্ઌͷใुΛߟྀ͢Δ Advantageɼ‫ࡏݱ‬ͷঢ়ଶʹ͓͚Δߦಈબ୒ͱߦಈΛධՁΛಉ࣌ʹߦ͏ Actor−Critic ͷ 3 ͭΛ߹Θͤͨ‫ڧ‬Խֶशख๏Ͱ
͋Δɽ
Trust Region Policy Optimization(TRPO)[4]ɿߋ
৽લͱߋ৽‫ޙ‬ͷํࡦ πθold ͱͷ KL μΠόʔδΣϯεΛ͖͍͠஋ δ ҎԼʹ཈͑Δ੍໿Λ༻͍ͯɼํࡦΛվળ͢Δ‫ڧ‬Խֶशख๏Ͱ͋Δɽ
Proximal Policy Optimization(PPO)[5]ɿTRPO Λ‫৽ߋʹج‬લͱߋ৽‫ޙ‬ͷํࡦ πθold ͷมԽྔΛҰఆͷൣғ಺ʹΫϦοϐϯά͜ͱͰֶशΛ҆ఆͤ͞Δ‫ڧ‬Խֶशख๏Ͱ͋Δɽ

3. Ձ஋ɾํࡦϕʔεͷ༗ޮੑௐࠪ
ෳ਺ͷλεΫʹ͓͍ͯ‫ڧ‬Խֶशख๏ͷ܏޲Λௐࠪ͢Δɽຊ࣮‫Ͱݧ‬͸ɼ࢖༻͢Δ‫ڧ‬Խֶशख๏ͱͯ͠ɼDQNɾCategolical
DQNɾA3CɾTRPOɾPPO Λ༻͍ΔɽํࡦϕʔεRandomDQN
Categolical
DQN
A3C
TRPO
PPO

Breakout
2.2
89.2

Qbert
375.0
4325.6

Spaceinvaders
75.0
803.5

Ms.Pacman
218.4
4226.5

Pong
0.0
19.3

406.5

14099.5

1483.53995.2

21.0

2.5
2.6
427.5

4353.24401.2
15897.6

486.2
876.6
1128.5

1941.6
1698.21680.4

18.7
21.0
21.0

4. ͓ΘΓʹ
ຊ‫Ͱڀݚ‬͸ɼAtari 2600 ʹ͓͚Δ༷ʑͳ‫ڧ‬Խֶशख๏
ͷ༗ޮੑௐࠪΛߦͬͨɽใुΛ֫ಘͰ͖ͳ͍‫ʹڥ؀‬͸Ձ஋ϕʔεɼ൓ରʹৗʹҰఆͷใुΛ֫ಘͰ͖Δ‫ʹڥ؀‬͸
Categolical DQNɼPPO ͕༗ޮͰ͋Δ͜ͱ͕Θ͔ͬͨɽ͜
ͷ܏޲Λ΋ͱʹɼ‫ط‬ଘͷ‫ڧ‬Խֶशख๏Λ૊Έ߹ΘͤͨΞϯαϯϒϧ‫ڧ‬Խֶशख๏ͷ։ൃ͕‫ظ‬଴Ͱ͖Δɽࠓ‫ޙ‬͸ෳ਺ͷ‫ڧ‬Խֶशख๏ͷߋͳΔௐࠪΛߦ͏ɽࢀߟจ‫ݙ‬[1] V. Mnih, et al., “Playing Atari with Deep Reinforcement Learning”, NIPS Deep Learning Workshop,
2013.
[2] M. G.Bellemare, et al., “A Distributional Perspective on Reinforcement Learning”, ICML, 2017.
[3] V. Mnih, et al., “Asynchronous Methods for Deep
Reinforcement Learning”, ICML, 2016.
[4] J. Schulman, et al., “Trust Region Policy Optimization”, ICML, 2015.
[5] J. Schulman, et al., “Proximal Policy Optimization
Algorithms”, arXiv:1707.06347, 2017.

Hessian-Free ๏Λ༻͍ͨ Echo State Network ͷߏ଄୳ࡧࢦಋ‫ڭ‬तɿ౻٢߂࿱

ER17059 ᖛ஍ ༏ً
1. ͸͡Ίʹ
ਓؒͷ೴ʹ͸໾ׂʹΑΓ‫ػ‬ೳ͢ΔՕॴ͕෼͔Ε͍ͯ͘‫ػ‬ೳ෼Խͱ‫ݺ‬͹ΕΔ‫ݱ‬৅͕͋ΔɽҰํɼਓ޻χϡʔϥϧωοτϫʔΫͰ΋ɼEcho State Network (ESN)[1] Λ༻͍ͨ‫ػ‬ೳ෼Խͷख๏͕ఏҊ͞Ε͓ͯΓɼಛఆͷ࣌ؒ·ͨ͸ۭؒύλʔϯʹ൓Ԡ͢ΔχϡʔϩϯΛֶश͢Δ͜ͱ͕֬ೝ͞Εͯ
͍Δ [2]ɽ͜ͷख๏Ͱ͸ɼGenetic Algorithm (GA) Λ༻͍ͯ ESN ͷ Reservoir ૚ͷχϡʔϩϯͷߏ଄ͱॏΈΛ୳ࡧ
͢Δɽ୳ࡧͷ݁Ռ͔Β Reservoir ૚ͷχϡʔϩϯͷग़ྗͱਖ਼ղϥϕϧ͔Β૬‫ޓ‬৘ใྔΛࢉग़͠ɼ‫ػ‬ೳ෼ԽΛ֬ೝͯ͠
͍Δɽ͔͠͠ɼGA ʹΑΔ࠷దͳॏΈͷ୳ࡧʹ͸๲େͳ͕࣌ؒඞཁʹͳΔɽͦ͜Ͱຊ‫Ͱڀݚ‬͸ɼHessian-Free ๏Λ༻͍ͨΑΓޮ཰తͳ୳ࡧํ๏ΛఏҊ͢Δɽ

2. ઌߦ‫ڀݚ‬
ESN ͸ɼೖྗ૚ɼReservoir ૚ɼग़ྗ૚ͷ 3 ૚Ͱߏ੒͞
Εɼग़ྗ૚ͷΈΛֶश͢Δɽೖྗ૚ͱ Reservoir ૚ͷॏΈ͸0 ͔Β 1 ͷؒͰϥϯμϜʹܾఆ͢ΔɽYamaguti Β [2] ͸ɼਤ 1 ͷΑ͏ʹ Reservoir ૚ͷχϡʔϩϯΛ Input Neurons
ͱ Output Neurons ʹ෼͚ͨ߆ଋ৚݅෇͖ ESN Ͱۭؒɾ࣌ؒύλʔϯͷ෼ྨΛߦ͍ͬͯΔɽग़ྗ૚ͷֶश‫ޙ‬ɼGA
Λ༻͍ͯ Reservoir ૚ͷߏ଄ͱॏΈΛ୳ࡧ͢Δɽͦͷࡍͷ֤χϡʔϩϯͷ૬‫ޓ‬৘ใྔ͔ΒɼωοτϫʔΫͷ‫ػ‬ೳ෼ԽΛ֬ೝ͍ͯ͠Δɽ͔͠͠ɼશχϡʔϩϯͷॏΈͷ૊Έ߹Θͤ͸๲େͳͨΊɼGA ʹΑΔ࠷దͳߏ଄ͱॏΈͷ྆ऀΛे෼ʹ୳ࡧ͢Δ͜ͱ͸ࠔ೉Ͱ͋Δɽ
Reservoir層
Input
Output𝑊𝑟𝑒𝑠 Neurons
Neurons

出力層
𝑠𝑝

𝑊𝑜𝑢𝑡

𝑊𝑖𝑛
入力層

૚ͷχϡʔϩϯͷ઀ଓߏ଄Λ୳ࡧ͢Δɽstep3 Ͱ͸ɼGA
ʹΑͬͯੜ੒͞ΕΔ‫ݸ‬ମͷਫ਼౓͕ऩଋ͢Δ·Ͱɼstep1 ͱ
step2 Λ‫܁‬Γฦ͢ɽਤ 2 ͷΦϨϯδ৭ͷ໼ҹ͸୳ࡧʹΑΓ༏Ε͍ͯΔͱൃ‫͞ݟ‬Εͨߏ଄Λࣔ͢ɽ
STEP 3
STEP１

STEP２

出力層

Reservoir層

入力層

Input Neurons Output Neurons

Hessian-Free法
による重みの学習

4. ࣮‫֓ݧ‬ཁ

出力層
Reservoir層GAよる
構造探索後 入力層

Input Neurons Output Neurons

GAの探索による
優れた構造

ਤ 2. ESN ͷߏ଄୳ࡧͷྲྀΕ

Reservoir ૚ͷߏ଄ͱॏΈͷ୳ࡧʹ͍ͭͯɼैདྷख๏ͱ
ఏҊख๏ͷൺֱ࣮‫ݧ‬Λߦ͏ɽ࣮‫ʹݧ‬͸ Reservoir ૚ʹ 32 ‫ݸ‬ͷχϡʔϩϯΛ࣋ͭ߆ଋ৚݅෇͖ ESN Λ༻͍ͯɼcos ೾ͷप೾਺ͱ෼ׂ਺Λ૊Έ߹Θͤͨύλʔϯͷ෼ྨΛߦ͏ɽఏҊख๏ʹ͓͚Δ Hessian-Free ๏ͷֶशճ਺͸ 20Epoch Ͱ͋ΔɽGA ͷύϥϝʔλ͸ɼ‫ݸ‬ମ਺Λ 200ɼ౫ଡ͢Δ਺Λ
180ɼੈ୅਺Λैདྷख๏Ͱ 500 ੈ୅ɼఏҊख๏Ͱ 5 ੈ୅ͱ͢ΔɽGA ʹΑΔ୳ࡧ‫ޙ‬ɼఏҊख๏ͱैདྷख๏Ͱ Reservoir
૚ͷχϡʔϩϯͱਖ਼ղϥϕϧ͔Β૬‫ޓ‬৘ใྔΛ‫ٻ‬Ί‫ػ‬ೳ෼ԽΛ֬ೝ͢Δɽ

5. ࣮‫݁ݧ‬Ռ
ද 1 ʹਫ਼౓ൺֱͷ݁ՌΛࣔ͢ɽද 1 ΑΓ࣌ؒ৘ใͱۭؒ
৘ใͷ྆ํͰఏҊख๏ͷਫ਼౓͕ߴ͍ɽ͜ΕʹΑΓɼHessianFree ๏Λ༻͍Δ͜ͱͰߏ଄ͱॏΈͷޮ཰తͳ୳ࡧ͕ՄೳͰɹ͋Δɽ
ද 1. ैདྷख๏ͱఏҊख๏ͷൺֱਫ਼౓ [%]
෼ྨछྨ

𝑡𝑝
𝑊𝑜𝑢𝑡

ਤ 1. ߆ଋ৚݅෇͖ ESN ͷωοτϫʔΫߏ଄

3. ఏҊख๏
ຊ‫Ͱڀݚ‬͸ɼχϡʔϩϯͷॏΈ࠷దԽͱ ESN ͷߏ଄୳ࡧΛஈ֊తʹߦ͏ख๏ΛఏҊ͢ΔɽશχϡʔϩϯͷॏΈΛ
Hessian-Free ๏ʹΑΔॏΈͷ࠷దԽͱɼGA ʹΑΔ Reservoir ૚ͷߏ଄Λ୳ࡧΛ‫܁‬Γฦ͢͜ͱͰɼޮ཰తʹ୳ࡧ͢Δ
͜ͱ͕ՄೳͱͳΔɽ

ैདྷख๏

ఏҊख๏

ɹ ɹ ۭؒ৘ใ

85.6

100.0

࣌ؒ৘ใ

55.3

91.2

ਤ 3(a) ʹैདྷख๏, ਤ 3(b) ʹఏҊख๏ͷ࣌ؒͱۭؒͷ૬‫ޓ‬৘ใྔͷ݁ՌΛࣔ͢ɽਤ 3(a) ͔Βɼۭؒɼ࣌ؒ৘ใྔ
ͷ྆ऀͱ΋ʹߴ͘ͳΔΑ͏ʹਐԽ͍ͯ͠Δ͜ͱ͕෼͔ΔɽҰํɼఏҊख๏Ͱ͸ۭؒɼ࣌ؒ৘ใͷͲͪΒ͔͕ߴ͘ͳΔΑ͏ʹਐԽ͍ͯ͠ΔɽҎ্ΑΓɼఏҊख๏͸ɼχϡʔϩϯͷ‫ػ‬ೳ෼ԽΛ༠ൃ͢Δ͜ͱ͕ՄೳͰ͋Δɽ

3.1.Hessian-Free ๏Λ༻͍ͨ ESN ͷֶश
Hessian-Free ๏͸ɼ࠶‫ܕؼ‬χϡʔϥϧωοτϫʔΫͷֶशʹ͓͚Δޯ഑ফࣦɾരൃ໰୊Λղܾͭͭ͠ɼ௕‫ظ‬తͳσʔλͷಛ௃நग़͕Մೳͳֶश๏Ͱ͋Δ [3]ɽHessian-Free ๏͸χϡʔϩϯͷॏΈͷߋ৽஋͕ᮢ஋ҎԼͷ࣌͸ॏΈͷ୳ࡧΛ‫ڞ‬໾ޯ഑๏Ͱߦ͏ɽ‫ڞ‬໾ޯ഑๏Λ༻͍ͨ৔߹ͷॏΈͷߋ৽ࣜΛࣜ (1) ʹࣔ͢ɽ
Wk+1 = Wk − Tk Mk(1)
͜͜ͰɼWk ͸‫ࡏݱ‬ͷॏΈͷू߹Ͱ͋ΓɼWk+1 ͸ߋ৽‫ޙ‬ͷߋ৽‫ޙ‬ͷॏΈͷू߹Ͱ͋Δɽࣜ (1) ͷ Tk ͸઀ϕΫτϧɼMk ͸‫ڞ‬໾ޯ഑Ͱ͋ΔɽҰํͰɼߋ৽஋͕ᮢ஋Ҏ্ͷ࣌͸ॏΈͷ୳ࡧΛΨ΢εχϡʔτϯ๏Ͱߦ͏ɽΨ΢εχϡʔτϯ๏Λ༻͍ͨ৔߹ͷॏΈͷߋ৽ࣜΛࣜ (2) ʹࣔ͢ɽ
Wk+1 = Wk − (∇FkT ∇Fk )−1 Fk ∇Fk(2)͜͜Ͱɼࣜ (2) ͷ Fk ͸‫ࠩޡ‬ͷ஋ɼ∇Fk ͸ޯ഑ϕΫτϧͰ
͋Δɽ

3.2.GA ʹΑΔߏ଄ͷ୳ࡧ
Reservoir ૚ͷχϡʔϩϯͷߏ଄Λܾఆ͢ΔͨΊʹɼGA
ʹΑΔ୳ࡧΛߦ͏ɽਤ 2 ʹఏҊख๏ʹΑΔߏ଄୳ࡧͷྲྀΕΛࣔ͢ɽ·ͣɼstep1 ʹͯɼશχϡʔϩϯͷॏΈΛ HessianFree ๏Ͱ࠷దԽ͢Δɽstep2 ͸ɼGA Λ༻͍ͯ Reservoir(a) ैདྷख๏

ਤ 3. ‫ػ‬ೳ෼Խͷ݁Ռ(b) ఏҊख๏

6. ͓ΘΓʹ
ຊ‫Ͱڀݚ‬͸ޮ཰తʹॏΈΛܾఆ͢Δํ๏ΛఏҊ͠ɼগͳ͍ੈ୅਺Ͱ‫ػ‬ೳ෼ԽΛ༠ൃ͢Δ͜ͱ͕ग़དྷͨɽࠓ‫ޙ‬ͷ༧ఆͱͯ͠͸ɼ୳ࡧ‫ޙ‬ͷ Reservoir ૚ͷχϡʔϩϯ͕࣌ؒͱۭ
ؒύλʔϯͷ৘ใΛͲͷΑ͏ʹॲཧ͍ͯ͠Δͷ͔ʹ͍ͭͯͷ෼ੳɼߟ࡯ͳͲ͕‫͛ڍ‬ΒΕΔɽ

ࢀߟจ‫ݙ‬[1] H. Jaeger, et al., “Harnessing Nonlinearity: Predicting Chaotic Systems and Saving Energy in WirelessCommunication”, Scienceɼ2004.
[2] Y. Yamaguti, et al., “Functional diﬀerentiations inevolutionary reservoir computing networks”, Chaos
2021.
[3] J. Martens, et al., “Learning recurrent neural networks with hessian-free optimization”, ICML, 2011.