NISTIR832Review

[LinktoNSTIR832document]

● InSection4,Explanation(asapillar)hassomenuancethatismissed:

○ ExplanationQuality

■ Accuracyofexplanations,inparticular,ShapleyValueestimations

(seeSectionVof[Datta,Sen,Zick2016]).SeeSundararajan&Najmi

2020foradditionalnuances,includingthefactthatcertaininstances

oftheSHAPlibrarydoesnotevenestimateShapleyValues.

■ Capturingcausalinfluenceinlocalexplanations,i.e.identifying

featuresthataretrulydrivingthemodel’spredictionsandteasing

themapartfromassociatedfeatures.

■ The“meaningful”pillarshouldincorporateanotionofsufficiency.

Explanationsmustbeunderstandablebutalsosensicalandenough

tojustifythepredictedoutcome.Wemustleverageimportantinput

andinternalfactorsasawaytoevaluatesufficiencyofexplanations

[Leinoetal2018,Wangetal.2020,Luetal.ACL2020].

○ Privacy-preservingexplanations:

■ Explanationsmustalsoideallyretainprivacy,inthatwedonot

disclosesensitiveinformationaboutindividualswhenjustifyinga

prediction.See[Datta,Sen,Zick2016]forprivacy-preserving

explanationsforShapleyValuefeatureimportances.Thisisalsoa

challengeforcounterfactualexplanationsandactionablerecourse

[Karimietal2020].

● Theprovideddefinitionofglobalexplanationsistoonarrow-- itismorethanthe

abilitytoproduceamodelthatexplains/approximatestheunderlyingmodel.Global

explanationsliketheexamplescitedinthepaper(SHAP,TCAV,ICE,PDPs)and

alsootherwork(seehere)providegeneralvisualizationsormetricsthat

characterizemodeldriversoverallorinsegments.

○ Alotofthesamequalityrequirementsthatarenecessaryforper-decision

examplesalsoapplyglobalexplanationssuchas:1)beingcausallyrelevant

tothebehaviorofthemodel,2)providingperfeatureexplanations.

○ Itisimportantforglobalandper-decisioninterpretabilitymethodstobe

consistent.Forexample,itshouldnotbethecasethatafeaturethatis

globallyimportantisunimportantforanydecisioninparticular.

● Stabilityasacorecomponent:Intheintro(andbeyond)theydefineAIas

“resilient”whichismoreofasecurityconcern(e.g.resilienttoadversarialattacks).

Anothercorecomponentwhichisignoredis“stability”-- understandingthatpeople

anddatachangesandmodelsmustberobusttothisorchangeaswell.[SR-11-7]
 

 

  

 

 

  ​ 

 

  

 

 

 

 

   

 

 

   

 

  

 

   

 

 

 

 

     

 

 

   

 

 

  

      

   

 

 

 

○ Whilestabilitymightbewell-definedfromacomputerscienceandstatistics

perspective,italsoplaysaroleinthepsychologyofhowpeopletrust

models.

● Nonblack-boxmodelslikeneuralnetworkswilloftendobetterthantheir

whiteboxcounterparts.Thisisn’tbroughtupinSection5.Itseemsunfairtosay

thatwhiteboxmodelsareidealfortrustworthyAI-- whiletheymightbemore

interpretable,theyalsomighthavepoorknowledge limits. Asanexample,deep

learningmodelsarehigh-dimensionalfunctionsandcancapturenuancesabout

datathatcouldactuallyprevent biasormakethemodeldecisionmorerobust.It’s

notjustaccuracyvs.interpretabilitybutaccuracyvs.fairnessvs.stabilityvs.

interpretabilityetc.

○ InSection5.1,Shapleyvaluesareacasewhereyoucangetaccuratemodel

explanationsevenifthemodelisablackbox.

○ Thereisagradientofblackbox->whiteboxmodels.Asanexample,In

Section5.1,theclaimthatGA2Misawhiteboxmodelisnotentirely

truthful.Whiletheycontainpairwiseinteractionsthatareessentially

heatmaps,it’shardtointerprettheactualfeaturevalueofthese

interactions.

● Wouldsuggestaddinginformationoncalibration(w.r.t.themetacognitionpoint).

Insection6.4,thepointtheybringuponmetacognitionhasbeenstudiedinthe

statisticsandmachinelearningliteratureascalibration[Niculescu-Mizil&Caruana,

Jiangetal.etc].

● Wouldsuggestaddinginformationonactionablerecoursewithinthe

counterfactualexplanationssection-- thebenefitofcounterfactualsarethatyou

arestress-testingmodelsonmodifieddatapoints.Thistiesindirectlytothe

psychologyofusingtrustworthyAI-- playingoutwhat-ifscenarios,andallowing

laypeopletounderstandhowdecisionscanbechanged.Somecitations:[Rawal&

Lakkaraju,2020;Karimiet.al2020;Poyiadziet.al2020,etc.]

● InSection6.2andSection6.3,itisnotenoughforhumanstobeabletousemodel

explanationstojustifyamodeldecision.Instead,ifgivenaccesstomodel

explanations,humansshouldbeabletoreplicatethemodeldecision-- i.e.come

tothesameconclusionontheirown.Thispreventshumansfromimplicitlytrusting

amodelandthenretroactivelytryingtojustifyit,whichisamajorproblemthatis

discussed.Thistiesdirectlyintothesufficiencyofexplanationsmentioned

earlier;see[Leinoetal2018,Wangetal.CVPRWorkshop2020,Luetal.ACL

2020].

● We’dliketoseebiasandfairnessmentionedin6.4.Knowingyour“knowledge

limits”alsomeansunderstandingwhatconscious/unconsciousbiasesyoupossess

andtryingtoactivelyundothatwhenmakingadecision.Humansdothisallthe

time,andmodelsneedtodothesame.6.4
 

 

 

 

 

      

○ Thisisnotassimpleastheexamplestheygaveofmodel

confidence/identifyingout-of-distributionpoints.Youcould(confidently)

reflecthumanbiasesinamodelevenwithabundantdata.

○ Weashumanshavepredefinedprotectedclassesthatyoucannot

discriminateon.Quantifyingdisparateimpactshouldbeaprerequisiteto

adoptionofanyAI.[Datta,Sen,Zick2016;Feldmanetal2016;Duttaetal

2020]
 

  ​ 

 

 ​ 

 

  ​ 

 

 ​ 

 

  ​ 

 

 ​ 

 

 

  ​ 

 

  ​ 

References

Dutta,Sanghamitra,etal.“FairnessUnderFeatureExemptions:Counterfactualand

ObservationalMeasures.”ArXiv:2006.07986 [Cs, Math, Stat],June2020.

arXiv.org,http://arxiv.org/abs/2006.07986.

Feldman,Michael,etal.“CertifyingandRemovingDisparateImpact.”ArXiv:1412.3756

[Cs, Stat],July2015.arXiv.org,http://arxiv.org/abs/1412.3756.

Karimi,Amir-Hossein,etal.“ASurveyofAlgorithmicRecourse:Definitions,

Formulations,Solutions,andProspects.”ArXiv:2010.04050 [Cs, Stat],Oct.2020.

arXiv.org,http://arxiv.org/abs/2010.04050.

Leino,Klas,etal.“Influence-DirectedExplanationsforDeepConvolutionalNetworks.”

ArXiv:1802.03788 [Cs, Stat],Nov.2018.arXiv.org,

http://arxiv.org/abs/1802.03788.

Lu,Kaiji,etal.“InfluencePathsforCharacterizingSubject-VerbNumberAgreementin

LSTMLanguageModels.”ArXiv:2005.01190 [Cs],May2020.arXiv.org,

http://arxiv.org/abs/2005.01190.

Poyiadzi,Rafael,etal.“FACE:FeasibleandActionableCounterfactualExplanations.”

Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society,Feb.2020,

pp.344–50.arXiv.org,doi:10.1145/3375627.3375850.

Rawal,Kaivalya,andHimabinduLakkaraju.“InterpretableandInteractiveSummaries

ofActionableRecourses.”ArXiv:2009.07165 [Cs, Stat],Sept.2020.arXiv.org,

http://arxiv.org/abs/2009.07165.

Sundararajan,Mukund,andAmirNajmi.“TheManyShapleyValuesforModel

Explanation.”ArXiv:1908.08474 [Cs, Econ],Feb.2020.arXiv.org,

http://arxiv.org/abs/1908.08474.
 

  ​ 

Wang,Zifan,etal.“InterpretingInterpretations:OrganizingAttributionMethodsby

Criteria.”ArXiv:2002.07985 [Cs],Apr.2020.arXiv.org,

http://arxiv.org/abs/2002.07985.