஌ࣝৠཹʹ͓͚Δ Multiple Teacher Network ͷબ୒ࢦಋ‫ڭ‬तɿ౻٢߂࿱

ER17048 தౡ྅༎1. ͸͡Ίʹ஌ࣝৠཹ͸ Teacher Network ͷ஌ࣝΛ Student Networkʹ఻͑Δ͜ͱͰɼਫ਼౓௿ԼΛ཈੍ͭͭ͠ϞσϧαΠζΛѹॖ͢Δख๏Ͱ͋Δ. Teacher Network ͱ Student Network ͷϞσϧαΠζʹେ͖ͳΪϟοϓ͕͋ΔͱɼStudent Networkͷਫ਼౓͕޲্͠ͳ͍໰୊͕͋Δɽຊ‫Ͱڀݚ‬͸ɼαΠζͷҟ
ͳΔෳ਺ͷ Teacher Network Λ༻͍Δ Multiple TeacherNetwork ΛఏҊ͠ɼ஌ࣝৠཹͷߴਫ਼౓ԽΛ໨తͱ͢Δɽ

2. ஌ࣝৠཹͱ໰୊఺஌ࣝৠཹʹ͸ɼTeacher Network ͷग़ྗΛར༻͢Δ Knowledge Distillation (KD)[1] ͱɼதؒ৘ใΛར༻͢Δ Contrastive Representation Distillation(CRD)[2] ͕͋Δɽද
1 ʹ Student Network Λ ResNet20 ʹͨ͠৔߹ͷ KD ʹΑΔৠཹ‫ޙ‬ͷਫ਼౓ͱ Teacher Network ͷϞσϧαΠζ (ύϥϝʔλ਺) Λࣔ͢ɽTeacher Network ͕ WRN40 2 ΑΓେ͖ͳϞσϧαΠζʹͳΔͱɼେ෯ͳਫ਼౓௿ԼΛট͘͜ͱ͕Θ͔Δɽຊ‫Ͱڀݚ‬͸ɼ͜ͷ໰୊͕ൃੜ͢Δ Teacher NetworkΛ Large Teacher Network(LTN)ɼൃੜ͠ͳ͍ Teacher
Network Λ Medium Teacher Network(MTN) ͱ‫ݺ‬শ͢Δɽ͜ͷ໰୊Λղܾ͢Δख๏ͱͯ͠ɼTeacher Network ͱStudent Network ؒʹ Teacher Assistant(TA) ͱ‫Ϳݺ‬ϞσϧΛ഑ஔ͠ɼஈ֊తʹৠཹ͢Δख๏ [3] ͕ఏҊ͞Ε͍ͯΔ͕ɼஈ֊తͳֶशΛඞཁͱ͢Δɽ·ͨɼ࠷ऴతͳଛࣦ͸ࣜ (3) ʹࣔ͢௨ΓɼશͯͷଛࣦΛ૯࿨ͨ͠஋ʹͳΔɽ
n#Ltotal =LiKDorCRD(3)
i=04. ධՁ࣮‫ݧ‬ఏҊख๏ͷ༗ޮੑΛධՁ͢ΔͨΊɼධՁ࣮‫ݧ‬Λߦ͏ɽTAͱ΋ൺֱ͠ධՁ͢Δɽ

4.1. ࣮‫֓ݧ‬ཁσʔληοτʹ͸ CIFAR-100 Λ࢖༻͠ɼֶश࣌ͷΤϙοΫ਺͸ 200ɼόοναΠζ͸ 64 ͱ͢ΔɽֶशϞσϧ͸ Student Network ʹ ResNet20ɼLTN ʹ WRN40 2ʙ
WRN28 10ɼMTN ʹ WRN28 1ʙWRN28 2 Λ༻͍Δɽ

4.2. ࣮‫݁ݧ‬Ռද 2 ͸ 2 ͭͷ Teacher Network Λ༻͍ͯৠཹͨ͠ਫ਼౓Λࣔ͢ɽTN1ɼ2ɼ3 ʹ͸ͦΕͧΕҟͳΔϞσϧΛ࢖༻͠ɼTN3 ͸ TN2 ΍ TN1 ΑΓେ͖͘ɼTN2 ͸ TN1 ΑΓେ͖͍ϞσϧͰ͋ΔɽTeacher Network ͷ૊Έ߹Θͤ͸ɼMTNͷΈɼMTN+LTNɼLTN ͷΈͷ 3 छྨ͋Δɽද 2: 2 ͭͷ Teacher Network Ͱৠཹͨ͠ਫ਼౓ఏҊख๏Teacher NetworkTN1TN2ResNet32

WRN28 271.12 71.26 72.13

Teacher Network WRN28 1 ResNet32 WRN28 2 WRN40 2 WRN28 6 WRN28 10

MTN+LTN ResNet32

WRN28 670.83 71.24 71.73

ύϥϝʔλ਺

371,347

466,457 1,147,251 2,248,991 13,155,603 36,497,171

MTN+LTN ResNet32 WRN28 10 70.24 71.37 71.61

ਫ਼౓71.2771.12LTN ͷΈද 1: ResNet20 ʹର͢Δৠཹ‫ޙ‬ਫ਼౓

70.98

70.25

MTN ͷΈ70.14

69.60

3. ఏҊख๏
ຊ‫Ͱڀݚ‬͸ɼਤ 1 ʹࣔ͢Α͏ʹෳ਺ͷ Teacher Network͔Βৠཹ͢Δ Multiple Teacher Network ΛఏҊ͢Δɽ͜ͷͱ͖ɼͲͷΑ͏ʹ Teacher Network Λબ୒͢ΔͱޮՌతͰ͋Δ͔ௐࠪ͢ΔɽఏҊख๏Ͱ͸ɼTeacher Network ͷ૊Έ߹ΘͤΛ MTN+LTN ͱ͠ɼMTN ʹ Student Network
ͱ LTN ͷ‫͠౉ڮ‬తͳ໾ׂΛ‫ظ‬଴͢Δɽ·ͨɼMTN ͷΈͰৠཹΛߦͬͨ৔߹ΑΓ΋ɼLTN Λ‫ؚ‬Ίͯৠཹ͢Δ͜ͱͰɼͦͷ஌ࣝ΋‫͖Ͱ༻׆‬Δͱߟ͑Δɽ
ResNet32

WRN28_6

ResNet32

WRN28_6

WRN28_10

M

LM

LL

ਤ 1: ఏҊख๏ͷ֓೦ਤຊ‫Ͱڀݚ‬ͷ Student Network ͱ Teacher Network ؒͷৠཹख๏ʹ͸ɼKDɼCRD ͷ͍ͣΕ͔Λ༻͍ΔɽKD ͷଛࣦؔ਺Λࣜ (1) ʹࣔ͢ɽ

LKD = αLhard + (1 − α) τ 2 Lsof t(1)͜͜Ͱɼα ͸܎਺ɼτ ͸Թ౓ύϥϝʔλΛࣔ͢ɽCRD ͷଛࣦؔ਺Λࣜ (2) ʹࣔ͢ɽTSeg g /τ]T g S /τNge+M!
"T Seg g /τ+N Eq(gT ,gS |C=0) [1 − log]
NegT gS /τ + M
LCRD = Eq(gT ,gS |C=1) [log(2)͜͜Ͱɼτ ͸Թ౓ύϥϝʔλɼM ͸σʔληοτͷΧʔσΟφϦςΟͰɼσʔλͷछྨ͕Ͳͷ͘Β͍͋Δ͔ͷ౓߹͍Λද͍ͯ͠ΔɽN ͸ೖྗσʔλͷඇྨࣅϖΞͷ਺Ͱ͋Δɽg S
ͱ g T ͸֤ωοτϫʔΫͷग़ྗ૚खલ͔Βநग़ͨ͠ग़ྗΛઢ‫ܗ‬ม‫ͤ͞׵‬ɼL2 ϊϧϜͰਖ਼‫ن‬Խͨ͠΋ͷͰ͋Δɽ

KDCRD

WRN28 6 WRN28 10 69.37 69.92 70.75

ද 2 ͔ΒɼMTN ͷΈͷ৔߹͕࠷΋ਫ਼౓͕ߴ͘ɼLTN ͷΈͷ৔߹͕࠷΋௿͘ͳͬͨɽMTN+LTN ͷ৔߹͸ɼLTNͷΈΑΓਫ਼౓͕ߴ͍ɽҎ্ΑΓɼ2 ͭͷ Teacher Networkͷ͏ͪ 1 ͭΛ MTN ʹ͢Δ͜ͱͰɼStudent Network ͱ
LTN ؒͷৠཹͰੜ͡Δ໰୊Λճආ͢Δ͜ͱ͕Ͱ͖Δɽද 3: 3 ͭͷ Teacher Network Ͱৠཹͨ͠ਫ਼౓ఏҊख๏Teacher Network
TN1TN2TN3TA

KD

CRD

MTN ͷΈ WRN28 1 ResNet32 WRN28 2 71.18 71.92 72.19
MTN+LTN ResNet32 WRN28 2 WRN28 6 70.24 71.06 71.83
MTN+LTN WRN28 2 WRN28 6 WRN28 10 68.87 70.15 71.00
LTN ͷΈ: KD or CRD

TA

WRN40 2 WRN28 6 WRN28 10 68.99 70.01 70.55

ද 3 ͸ 3 ͭͷ Teacher Network Λ༻͍ͯৠཹͨ͠ਫ਼౓
Λࣔ͢ɽද 3 ͔ΒɼMTN+LTN ͷ৔߹ɼਫ਼౓௿ԼΛ཈੍͢Δ͜ͱ͕Ͱ͖ͳ͍͜ͱ͕͜ͱ͕Θ͔ΔɽMTN ͷΈͷ৔߹͕࠷΋ߴਫ਼౓Ͱ͋ΔɽҎ্ΑΓɼTeacher Network ਺Λ૿΍ͯ͠΋ LTN ͕͋Δ৔߹ɼߴਫ਼౓Խ͕೉͍͜͠ͱ͕൑໌ͨ͠ɽ

5. ͓ΘΓʹຊ‫Ͱڀݚ‬͸ɼೳྗࠩͷ͋Δ Student Network ͱ TeacherNetwork ؒͷৠཹͷࡍʹɼMTN+LTN ͷ૊߹ͤΛ‫ؚ‬ΊͯͦΕΒΛಉ࣌ʹৠཹͤ͞Δ͜ͱΛఏҊͨ͠ɽ͜ΕʹΑΓɼ
Teacher Network Λ 2 ͭ࢖༻͢Δ৔߹͸ਫ਼౓ͷ௿ԼΛ཈
͑ɼMTN ͷΈͷਫ਼౓ʹඖఢ͠ɼ͞Βʹ Teacher Network͕ 1 ͭͷΈͰৠཹΛߦͬͨ৔߹ΑΓ΋ਫ਼౓͕޲্͢Δ͜ͱ
Λ֬ೝͨ͠ɽ·ͨɼTA ͱҟͳΓஈ֊తͳֶशΛඞཁͱͤͣɼͦͷ෼ֶश࣌ؒΛ୹ॖͰ͖Δ͜ͱ΋֬ೝͨ͠ɽࢀߟจ‫ݙ‬[1] G. Hinton, et al., “Distilling the knowledge in a neural network”, NeurIPS Workshop, 2015.
[2] YɽTian, et al., “Contrastive Representation Distillation”, ICLR, 2020.
[3] S. Mirzadeh, et al., “Improved Knowledge Distillation via Teacher Assistant”, AAAI, 2020.