Bibliography

[AALC21] Emre ATES, Burak AKSAR, Vitus J LEUNG, and Ayse K COSKUN. « Counterfac-
tual Explanations for Multivariate Time Series ». In *2021 International Conference on
Applied Artificial Intelligence (ICAPAI)*, pages 1–8. IEEE, 2021. 1.2, 1.2.2

[AB18] Amina ADADI and Mohammed BERRADA. « Peeking inside the black-box: a survey on explainable artificial intelligence (XAI) ». *IEEE access*, 6:52138–52160, 2018. 1.1.1

[ABB<sup>+</sup>21] Gilles AUDEMARD, Steve BELLART, Louenas BOUNIA, Frédéric KORICHE, Jean-Marie LAGNIEZ, and Pierre MARQUIS. « On the Computational Intelligibility of Boolean Classifiers ». In *Proceedings of the 18th International Conference on Principles of Knowledge Representation and Reasoning*, pages 74–86, 11 2021. 1.2.1, 2.2, 2.2.1, 3.3.4

[ABB<sup>+</sup>22a] Gilles AUDEMARD, Steve BELLART, Louenas BOUNIA, Frédéric KORICHE, Jean-Marie LAGNIEZ, and Pierre MARQUIS. « Les raisons majoritaires: des explications abductives pour les forêts aléatoires ». *Extraction et Gestion des Connaissances: EGC'2022*, 38, 2022. 2.2, 2.2.2, 5

[ABB<sup>+</sup>22b] Gilles AUDEMARD, Steve BELLART, Louenas BOUNIA, Frédéric KORICHE, Jean-Marie LAGNIEZ, and Pierre MARQUIS. « On the explanatory power of Boolean decision trees ». *Data Knowledge Engineering*, Page 102088, 2022. 2.2

[ABD<sup>+</sup>18] Alekh AGARWAL, Alina BEYGELZIMER, Miroslav DUDÍK, John LANGFORD, and Hanna WALLACH. « A reductions approach to fair classification ». In *International Conference on Machine Learning*, pages 60–69. PMLR, 2018. 1.1.2

[ADLPR22] Nicholas ASHER, Lucas DE LARA, Soumya PAUL, and Chris RUSSELL. « Counterfac-
tual Models for Fair and Adequate Explanations ». *Machine Learning and Knowledge
Extraction*, 4(2):316–349, 2022. 2.1

[ADRDS<sup>+</sup>20] Alejandro Barredo ARRIETA, Natalia DÍAZ-RODRÍGUEZ, Javier DEL SER, Adrien BENNETOT, Siham TABIK, Alberto BARBADO, Salvador GARCÍA, Sergio GIL-LÓPEZ, Daniel MOLINA, Richard BENJAMINS, and OTHERS. « Explainable Artificial Intel-
ligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible
AI ». *Information fusion*, 58:82–115, 2020. 1, 1.1

[ADWF15] Babak ALIPANAH, Andrew DELONG, Matthew T WEIRAUCH, and Brendan J FREY. « Predicting the sequence specificities of DNA-and RNA-binding proteins by deep learning ». *Nature biotechnology*, 33(8):831–838, 2015. 1.1
[AET18] Muhammad Aurangzeb AHMAD, Carly ECKERT, and Ankur TEREDESAI. « Interpretable machine learning in healthcare ». In *Proceedings of the 2018 ACM international conference on bioinformatics, computational biology, and health informatics*, pages 559–560, 2018. 2.3

[AFM02] Jérôme AMILHASTRE, Hélene FARGIER, and Pierre MARQUIS. « Consistency restoration and explanations in dynamic CSPs—application to configuration ». *Artificial Intelligence*, 135(1-2):199–234, 2002. 2.2.1

[AGM$^{+}$18] Julius ADEBAYO, Justin GILMER, Michael MUELLY, Ian GOODFELLOW, Moritz HARDT, and Been KIM. « Sanity checks for saliency maps ». *Advances in neural information processing systems*, 31, 2018. 2.3

[AJ18] David ALVAREZ-MELIS and Tommi S. JAAKKOLA. « On the Robustness of Interpretability Methods ». *CoRR*, abs/1806.08049, 2018. 2.3

[AJM12] Muhammad Naeem AYYAZ, Imran JAVED, and Waqar MAHMOOD. « Handwritten character recognition using multiclass svm classification with hybrid feature extraction ». *Pakistan Journal of Engineering and Applied Sciences*, 2012. 1.1

[AKM20a] Gilles AUDEMARD, Frédéric KORICHE, and Pierre MARQUIS. « On tractable XAI queries based on compiled representations ». In *Proceedings of the International Conference on Principles of Knowledge Representation and Reasoning*, volume 17, pages 838–849, 2020. 2.2, 2.2.1, 3.3.4

[AKM20b] Gilles AUDEMARD, Frédéric KORICHE, and Pierre MARQUIS. « On Tractable XAI Queries based on Compiled Representations ». In *Proceedings of the 17th International Conference on Principles of Knowledge Representation and Reasoning*, pages 838–849, 2020. 1.2.2

[ALSA$^{+}$17] Elaine ANGELINO, Nicholas LARUS-STONE, Daniel ALABI, Margo SELTZER, and Cynthia RUDIN. « Learning certifiably optimal rule lists for categorical data ». *arXiv preprint arXiv:1704.01701*, 2017. 2.3, 5.2.2

[Alt92] Naomi S ALTMAN. « An introduction to kernel and nearest-neighbor nonparametric regression ». *The American Statistician*, 46(3):175–185, 1992. 1.1

[AMJ18] David ALVAREZ MELIS and Tommi JAAKKOLA. « Towards robust interpretability with self-explaining neural networks ». *Advances in neural information processing systems*, 31, 2018. 1.3

[ANORC13] Ignasi ABÍO, Robert NIEUWENHUIS, Albert OLIVERAS, and Enric RODRÍGUEZ-CARBONELL. « A parametric approach for smaller and better encodings of cardinality constraints ». In *International Conference on Principles and Practice of Constraint Programming*, pages 80–96. Springer, 2013. 3.2.1

[AP94] Agnar AAMODT and Enric PLAZA. « Case-based reasoning: Foundational issues, methodological variations, and system approaches ». *AI communications*, 7(1):39–59, 1994. 1.2.2

[APR21] Nicholas ASHER, Soumya PAUL, and Chris RUSSELL. « Fair and adequate explanations ». In *International Cross-Domain Conference for Machine Learning and Knowledge Extraction*, pages 79–97. Springer, 2021. 2.2
[ARAB$^{+}$17] P ALLISY-ROBERTS, P AMBROSI, DT BARTLETT, BM COURSEY, LA DEWERD, E FANTUZZI, and JC MCDONALD. « The 11th Annual State of Agile Report ». *Journal of the ICRU*, 6(2):7–8, 2017. 1.1

[ARLG20] Kasun AMARASINGHE, Kit T. RODOLFA, Hemank LAMBA, and Rayid GHANI. « Explainable Machine Learning for Public Policy: Use Cases, Gaps, and Research Directions ». *CoRR*, abs/2010.14374, 2020. 2.3

[AS$^{+}$94] Rakesh AGRAWAL, Ramakrishnan SRIKANT, and OTHERS. « Fast algorithms for mining association rules ». In *Proc. 20th int. conf. very large data bases*, VLDB, volume 1215, pages 487–499. Citeseer, 1994.

[AZ20] Daniel W APLEY and Jingyu ZHU. « Visualizing the effects of predictor variables in black box supervised learning models ». *Journal of the Royal Statistical Society: Series B (Statistical Methodology)*, 82(4):1059–1086, 2020. 2.1

[BAL$^{+}$21] Clara BOVE, Jonathan AIGRAIN, Marie-Jeanne LESOT, Charles TIJUS, and Marcin DETYNIECKI. « Contextualising local explanations for non-expert users: an XAI pricing interface for insurance. ». In *IUI Workshops*, 2021. 1.2.3

[BAL$^{+}$22] Clara BOVE, Jonathan AIGRAIN, Marie-Jeanne LESOT, Charles Albert TIJUS, and Marcin DETYNIECKI. « Contextualization and Exploration of Local Feature Importance Explanations to Improve Understanding and Satisfaction of Non-Expert Users ». *27th International Conference on Intelligent User Interfaces*, 2022.

[BB03] Olivier BAILLEUX and Yacine BOUFKHAD. « Efficient CNF encoding of boolean cardinality constraints ». In *International conference on principles and practice of constraint programming*, pages 108–122. Springer, 2003. 3.2.1

[BB13] Richard A BERK and Justin BLEICH. « Statistical procedures for forecasting criminal behavior: A comparative assessment ». *Criminology & Pub. Pol’y*, 12:513, 2013. 1.1

[BBM$^{+}$15] Sebastian BACH, Alexander BINDER, Grégoire MONTAVON, Frederick KLAUSCHEN, Klaus-Robert MÜLLER, and Wojciech SAMEK. « On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation ». *PloS one*, 10(7):e0130140, 2015. (document), 1.2.2, 1.7, 1.2.2, 1.2.3, 5.1.1, 6.1

[BCAMT20] Ryma BOUMAZOUZA, Fahima CHEIKH-ALILI, Bertrand MAZURE, and Karim TABIA. « A Symbolic Approach for Counterfactual Explanations ». In *International Conference on Scalable Uncertainty Management*, pages 270–277. Springer, 2020. 6.6

[BCAMT21a] Ryma BOUMAZOUZA, Fahima CHEIKH-ALILI, Bertrand MAZURE, and Karim TABIA. « ASTERYX: A model-Agnostic SaT-basEd appRoaCh for sYmbolic and score-based eXplanations ». In *Proceedings of the 30th ACM International Conference on Information & Knowledge Management*, pages 120–129, 2021. 6.6

[BCAMT21b] Ryma BOUMAZOUZA, Fahima CHEIKH-ALILI, Bertrand MAZURE, and Karim TABIA. « A Model-Agnostic SAT-based Approach for Symbolic Explanation Enumeration ». In *Proceedings of the 23rd International Conference on Artificial Intelligence*, 2021.

[BČB18] Jaroslav BENDÍK, Ivana ČERNÁ, and Nikola BENEŠ. « Recursive online enumeration of all minimal unsatisfiable subsets ». In *International symposium on automated technology for verification and analysis*, pages 143–159. Springer, 2018. 3.1, 3.3.1
[BCC$^{+}$16] Mariusz BOJARSKI, Anna CHOROMANSKA, Krzysztof CHOROMANSKI, Bernhard FIRNER, Larry JACKEL, Urs MULLER, and Karol ZIEBA. « Visual backprop: visualizing cnns for autonomous driving ». *arXiv preprint arXiv:1611.05418*, 2, 2016. 1.2.2

[BCNM06] C BUCILUA, R CARUANA, and A NICULESCU-MIZIL. « Model compression, in proceedings of the 12 th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining ». *New York, NY, USA*, 3, 2006. 1.2.2

[BCPdI19] Alberto BLANCO, Arantza CASILLAS, Alicia PÉREZ, and Arantza Diaz de ILARRAZA. « Multi-label clinical document classification: Impact of label-density ». *Expert Systems with Applications*, 138:112835, 2019. 4.2

[BdSRM14] Flávia Cristina BERNARDINI, Rodrigo Barbosa da SILVA, Rodrigo Magalhães RODOVALHO, and Edwin B. Mitacc MEZA. « Cardinality and Density Measures and Their Influence to Multi-Label Learning Methods ». *Learning and Nonlinear Models*, 12:53-71, 2014. 1.2

[BDTK14] Fahiem BACCHUS, Jessica DAVIES, Maria TSIMPOUKELLI, and George KATSIRELOS. « Relaxation search: A simple way of managing optional clauses ». In *Proceedings of the AAAI Conference on Artificial Intelligence*, volume 28, 2014. 3.3.4

[Ber21] Leopoldo BERTOSSI. « Declarative approaches to counterfactual explanations for classification ». *Theory and Practice of Logic Programming*, pages 1-35, 2021. 4.6

[BFOS84] Leo BREIMAN, JH FRIEDMAN, RA OLSHEN, and CJ STONE. « Classification and regression trees. Wadsworth & Brooks ». *Cole Statistics/Probability Series*, 1984. 1.2.1

[BH17] Stefan BERNDORFER and Aron HENRIKSSON. « Automated diagnosis coding with combined text representations ». *Stud Health Technol Inform*, 235:201-205, 2017. 4.2

[BHvM09] Armin BIERE, Marijn HEULE, and Hans van MAAREN. *Handbook of satisfiability*, volume 185. IOS press, 2009. 3.2, 3.1, 3.3.1

[BJL09] Ruth MJ BYRNE and Philip N JOHNSON-LAIRD. « ‘If’ and the problems of conditional reasoning ». *Trends in Cognitive Sciences*, 13(7):282-287, 2009. 1.2.1

[BK15] Fahiem BACCHUS and George KATSIRELOS. « Using minimal correction sets to more efficiently compute minimal unsatisfiable sets ». In *International Conference on Computer Aided Verification*, pages 70-86. Springer, 2015. 3.1, 3.3.1, 3.3.4, 3.3.4

[BK16] Fahiem BACCHUS and George KATSIRELOS. « Finding a collection of MUSes incrementally ». In *International Conference on AI and OR Techniques in Constraint Programming for Combinatorial Optimization Problems*, pages 35-44. Springer, 2016. 3.1, 3.3.1

[BL03] Elazar BIRNBAUM and Eliezer L LOZINSKII. « Consistent subsets of inconsistent systems: structure and behaviour ». *Journal of Experimental & Theoretical Artificial Intelligence*, 15(1):25-46, 2003. 3.1, 3.3.4, 3.3.4

[BL22] Isabelle BLOCH and Marie-Jeanne LESOT. « Towards a Formulation of Fuzzy Contrastive Explanations ». *2022 IEEE International Conference on Fuzzy Systems (FUZZ-IEEE)*, pages 1-8, 2022. 1.2.2
[BLS<sup>+</sup>18] Geuntae BAE, Hojae LEE, Sunghoon SON, Doha HWANG, and Jongseok KIM. « Secure and robust user authentication using partial fingerprint matching ». In *2018 IEEE International Conference on Consumer Electronics (ICCE)*, pages 1–6. IEEE, 2018. 1.1

[BLSB04] Matthew R BOUTELL, Jiebo LUO, Xipeng SHEN, and Christopher M BROWN. « Learning multi-label scene classification ». *Pattern recognition*, 37(9):1757–1771, 2004. 1.2

[BMPS20] Pablo BARCELÓ, Mikaël MONET, Jorge PÉREZ, and Bernardo SUBERCASEAUX. « Model interpretability through the lens of computational complexity ». *Advances in neural information processing systems*, 33:15487–15498, 2020. 1.2.1

[BMS11] Anton BELOV and Joao MARQUES-SILVA. « Accelerating MUS extraction with recursive model rotation ». In *2011 Formal Methods in Computer-Aided Design (FMCAD)*, pages 37–40. IEEE, 2011. 3.3.4

[BMS12] Anton BELOV and Joao MARQUES-SILVA. « MUSer2: An efficient MUS extractor ». *Journal on Satisfiability, Boolean Modeling and Computation*, 8(3-4):123–128, 2012. 3.3.4

[BP19] Ismaël BAAJ and Jean-Philippe POLI. « Natural language generation of explanations of fuzzy inference decisions ». In *2019 IEEE international conference on fuzzy systems (FUZZ-IEEE)*, pages 1–6. IEEE, 2019. 2.3

[BRB18] Wieland BRENDEL, Jonas RAUBER, and Matthias BETHGE. « Decision-Based Adversarial Attacks: Reliable Attacks Against Black-Box Machine Learning Models ». In *6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings*. OpenReview.net, 2018. 1.1.2

[Bre01a] Leo BREIMAN. « Random forests ». *Machine learning*, 45(1):5–32, 2001. 1.2.2

[Bre01b] Leo BREIMAN. « Statistical modeling: The two cultures (with comments and a rejoinder by the author) ». *Statistical science*, 16(3):199–231, 2001. 1.2.1

[Bry86] Randal E BRYANT. « Graph-based algorithms for boolean function manipulation ». *Computers, IEEE Transactions on*, 100(8):677–691, 1986. 2.2.1

[BS05] James BAILEY and Peter J STUCKEY. « Discovery of minimal unsatisfiable subsets of constraints using hitting set dualization ». In *International Workshop on Practical Aspects of Declarative Languages*, pages 174–186. Springer, 2005. 3.3.4, 3.3.4

[BSH<sup>+</sup>10] David BAEHRENS, Timon SCHROETER, Stefan HARMELING, Motoaki KAWANABE, Katja HANSEN, and Klaus-Robert MÜLLER. « How to explain individual classification decisions ». *The Journal of Machine Learning Research*, 11:1803–1831, 2010. 5.1

[BTD<sup>+</sup>16] Mariusz BOJARSKI, Davide Del TESTA, Daniel DWORAKOWSKI, Bernhard FIRNER, Beat FLEPP, Prasoon GOYAL, Lawrence D. JACKEL, Mathew MONFORT, Urs MULLER, Jiakai ZHANG, Xin ZHANG, Jake ZHAO, and Karol ZIEBA. « End to End Learning for Self-Driving Cars ». *CoRR*, abs/1604.07316, 2016. 1.1

[BVKV<sup>+</sup>18] Reuben BINNS, Max VAN KLEEK, Michael VEALE, Ulrik LYNGS, Jun ZHAO, and Nigel SHADBOLT. « 'It's Reducing a Human Being to a Percentage' Perceptions of
Justice in Algorithmic Decisions». In Proceedings of the 2018 Chi conference on human factors in computing systems, pages 1–14, 2018. 1.1.2

[BWM20] Umang BHATT, Adrian WELLER, and José MF MOURA. «Evaluating and aggregating feature-based model explanations». arXiv preprint arXiv:2005.00631, 2020. 2.3

[BXS<sup>+</sup>20] Umang BHATT, Alice XIANG, Shubham SHARMA, Adrian WELLER, Ankur TALY, Yunhan JIA, Joydeep GHOSH, Ruchir PURI, José MF MOURA, and Peter ECKERSLEY. «Explainable machine learning in deployment». In Proceedings of the 2020 conference on fairness, accountability, and transparency, pages 648–657, 2020. 1.1.3, 5.1.1

[Byr19] Ruth MJ BYRNE. «Counterfactuals in Explainable Artificial Intelligence (XAI): Evidence from Human Reasoning.». In IJCAI, pages 6276–6282, 2019. 1.2.2

[CD03] H. CHAN and Adnan DARWICHE. «Reasoning about Bayesian Network Classifiers». In UAI, 2003. 2.2.1

[CD07] Mark CHAVIRA and Adnan DARWICHE. «Compiling Bayesian Networks Using Variable Elimination.». In IJCAI, volume 2443, 2007. 2.2.1

[CE13] Paulo CORTEZ and Mark J EMBRECHTS. «Using sensitivity analysis and visualization techniques to open black box data mining models». Information Sciences, 225:1–17, 2013. (document), 1.2.2, 1.11

[CG16] Tianqi CHEN and Carlos GUESTRIN. «Xgboost: A scalable tree boosting system». In Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining, pages 785–794, 2016.

[CGG<sup>+</sup>20] Gabriele CIRAVEGNA, Francesco GIANNINI, Marco GORI, Marco MAGGINI, and Stefano MELACCI. «Human-Driven FOL Explanations of Deep Learning.». In IJCAI, pages 2234–2240, 2020. 4.1, 4.3

[Che21] Shikun CHEN. «Interpretation of multi-label classification models using shapley values». CoRR, abs/2104.10505, 2021. 4.1, 6.1

[Chr94] H CHRISTOS. «PAPADIMITRIOU: Computational complexity». Addison-Wesley, 2(3):4, 1994. 3.1

[CK01] Amanda CLARE and Ross D KING. «Knowledge discovery in multi-label phenotype data». In European conference on principles of data mining and knowledge discovery, pages 42–53. Springer, 2001. 2

[CLG<sup>+</sup>15] Rich CARUANA, Yin LOU, Johannes GEHRKE, Paul KOCH, Marc STURM, and Noemie ELHADAD. «Intelligible models for healthcare: Predicting pneumonia risk and hospital 30-day readmission». In Proceedings of the 21st ACM SIGKDD international conference on knowledge discovery and data mining, pages 1721–1730, 2015. 1.1, 1.3

[CLP<sup>+</sup>18] Jiaoyan CHEN, Freddy LÉCUÉ, Jeff Z PAN, Ian HORROCKS, and Huajun CHEN. «Knowledge-based transfer learning explanation». In Sixteenth International Conference on Principles of Knowledge Representation and Reasoning, 2018. 1.2, 1.3

[CLR<sup>+</sup>18] Chaofan CHEN, Kangcheng LIN, Cynthia RUDIN, Yaron SHAPOSHNIK, Sijia WANG, and Tong WANG. «An interpretable model with globally consistent explanations for credit risk». arXiv preprint arXiv:1811.12615, 2018. 1.1
[CNC<sup>+</sup>16] Jie-Zhi CHENG, Dong NI, Yi-Hong CHOU, Jing QIN, Chui-Mei TIU, Yeun-Chung CHANG, Chiun-Sheng HUANG, Dinggang SHEN, and Chung-Ming CHEN. « Computer-aided diagnosis with deep learning architecture: applications to breast lesions in US images and pulmonary nodules in CT scans ». *Scientific reports*, 6(1):1–13, 2016. 1.1

[CNQ03] Gianpiero CABODI, Sergio NOCCO, and Stefano QUER. « Improving SAT-based bounded model checking by means of BDD-based approximate traversals ». In *2003 Design, Automation and Test in Europe Conference and Exhibition*, pages 898–903. IEEE, 2003. 3.2.1

[Coo71] Stephen A COOK. « The complexity of theorem-proving procedures ». In *Proceedings of the third annual ACM symposium on Theory of computing*, pages 151–158, 1971. 3.1, 3.3.4

[CRT98] Alessandro CIMATTI, Marco ROVERI, and Paolo TRAVERSO. « Automatic OBDD-based generation of universal plans in non-deterministic domains ». In *AAAI/IAAI*, pages 875–881, 1998. 2.2.1

[CS95] Mark CRAVEN and Jude SHAVLIK. « Extracting tree-structured representations of trained networks ». *Advances in neural information processing systems*, 8, 1995. 1.2.1, 1.2.2

[CSGD20a] Arthur CHOI, Andy SHIH, Anchal GOYANKA, and Adnan DARWICHE. « On Symbolically Encoding the Behavior of Random Forests ». In *3rd Workshop on Formal Methods for ML-Enabled Autonomous Systems (FoMLAS)*, 2020. 2.2.2

[CSGD20b] Arthur CHOI, Andy SHIH, Anchal GOYANKA, and Adnan DARWICHE. « On symbolically encoding the behavior of random forests ». *arXiv preprint arXiv:2007.01493*, 2020. 3.3.4

[CSHB18] Aditya CHATTOPADHAY, Anirban SARKAR, Prantik HOWLADER, and Vineeth N BALASUBRAMANIAN. « Grad-cam++: Generalized gradient-based visual explanations for deep convolutional networks ». In *2018 IEEE winter conference on applications of computer vision (WACV)*, pages 839–847. IEEE, 2018. 1.2, 1.2.2, 1.2.3, 1.3

[CSSD17] Arthur CHOI, Weijia SHI, Andy SHIH, and Adnan DARWICHE. « Compiling neural networks into tractable Boolean circuits ». *intelligence*, 2017. 2.2.1

[CT95] Zhi-Zhong CHEN and Seinosuke TODA. « The complexity of selecting maximal solutions ». *Information and Computation*, 119(2):231–239, 1995. 3.1, 3.3.4

[CV95] Corinna CORTES and Vladimir VAPNIK. « Support-vector networks ». *Machine learning*, 20(3):273–297, 1995. 1.1

[CVKRBA12] Binu P CHACKO, VR VIMAL KRISHNAN, G RAJU, and P BABU ANTO. « Handwritten character recognition using wavelet energy and extreme learning machine ». *International Journal of Machine Learning and Cybernetics*, 3(2):149–161, 2012. 1.1

[CW17] Nicholas CARLINI and David WAGNER. « Towards evaluating the robustness of neural networks ». In *2017 IEEE symposium on security and privacy (sp)*, pages 39–57. IEEE, 2017. 1.1.1, 1.1.2
[CW18] Nicholas CARLINI and David WAGNER. « Audio adversarial examples: Targeted attacks on speech-to-text ». In *2018 IEEE Security and Privacy Workshops (SPW)*, pages 1–7. IEEE, 2018. 1.1.1

[Dar99] Adnan DARWICHE. « Compiling knowledge into decomposable negation normal form ». In *IJCAI*, volume 99, pages 284–289. Citeseer, 1999. 2.2.1

[Dar01] Adnan DARWICHE. « Decomposable negation normal form ». *Journal of the ACM (JACM)*, 48(4):608–647, 2001. 2.2.1

[Dar11] Adnan DARWICHE. « SDD: A new canonical representation of propositional knowledge bases ». In *Twenty-Second International Joint Conference on Artificial Intelligence*, 2011. 2.2.1

[Dar20] Adnan DARWICHE. « Three Modern Roles for Logic in AI ». In *Proceedings of the 39th ACM SIGMOD-SIGACT-SIGAI Symposium on Principles of Database Systems*, PODS’20, Page 229–243, New York, NY, USA, 2020. Association for Computing Machinery. 1.2.2, 3, 4.3.1

[Das18] Jeffrey DASTIN. Amazon scraps secret AI recruiting tool that showed bias against women. In *Ethics of Data and Analytics*, pages 296–299. Auerbach Publications, 2018. 1.1

[dCF09] André CPLF de CARVALHO and Alex A FREITAS. « A tutorial on multi-label classification techniques ». *Foundations of computational intelligence volume 5*, pages 177–195, 2009. 1

[DCL+18] Amit DHURANDHAR, Pin-Yu CHEN, Ronny LUSS, Chun-Chen TU, Paishun TING, Karthikeyan SHANMUGAM, and Payel DAS. « Explanations based on the missing: Towards contrastive explanations with pertinent negatives ». *Advances in neural information processing systems*, 31, 2018. 1.2, 1.2.2, 3.3.3

[DF18] Julia DRESSEL and Hany FARID. « The accuracy, fairness, and limits of predicting recidivism ». *Science advances*, 4(1):eaa05580, 2018. 1.1

[DFL+18] Lieyun DING, Weili FANG, Hanbin LUO, Peter ED LOVE, Botao ZHONG, and Xi OUYANG. « A deep hybrid learning model to detect unsafe behavior: Integrating convolution neural networks and long short-term memory ». *Automation in construction*, 86:118–124, 2018. 1.1

[DG17] D DUA and C GRAFF. « UCI machine learning repository. University of California, School of Information and Computer Science », 2017.

[DGW18] Sanjeeb DASH, Oktay GUNLUK, and Dennis WEI. « Boolean decision rules via column generation ». *Advances in neural information processing systems*, 31, 2018. 1.1, 1.3

[DH22] Adnan DARWICHE and Auguste HIRTH. « On the (Complete) Reasons Behind Decisions ». *Journal of Logic, Language and Information*, pages 1–26, 2022. 1.2, 1.2, 1.3, 2.2, 5

[DHP+12] Cynthia DWORK, Moritz HARDT, Toniann PITASSI, Omer REINGOLD, and Richard ZEMEL. « Fairness through awareness ». In *Proceedings of the 3rd innovations in theoretical computer science conference*, pages 214–226, 2012. 1.2, 1.3
[DLM<sup>+</sup>22] Sanghamitra DUTTA, Jason LONG, Saumitra MISHRA, Cecilia TILLI, and Daniele MAGAZZENI. « Robust Counterfactual Explanations for Tree-Based Ensembles ». In *International Conference on Machine Learning*, pages 5742–5756. PMLR, 2022. 1.2, 1.3

[DLP<sup>+</sup>18] Yinpeng DONG, Fangzhou LIAO, Tianyu PANG, Hang SU, Jun ZHU, Xiaolin HU, and Jianguo LI. « Boosting adversarial attacks with momentum ». In *Proceedings of the IEEE conference on computer vision and pattern recognition*, pages 9185–9193, 2018. 1.1.2

[DLT<sup>+</sup>18] Hanjun DAI, Hui LI, Tian TIAN, Xin HUANG, Lin WANG, Jun ZHU, and Le SONG. « Adversarial attack on graph structured data ». In *International conference on machine learning*, pages 1115–1124. PMLR, 2018. 1.1.2

[DM02] Adnan DARWICHE and Pierre MARQUIS. « A knowledge compilation map ». *Journal of Artificial Intelligence Research*, 17:229–264, 2002. 2.2.1, 3.2.1

[DPB<sup>+</sup>19] Amit DHURANDHAR, Tejaswini PEDAPATI, Avinash BALAKRISHNAN, Pin-Yu CHEN, Karthikeyan SHANMUGAM, and Ruchir PURI. « Model Agnostic Contrastive Explanations for Structured Data ». *CoRR*, abs/1906.00117, 2019. 1.2, 1.2.2

[DR20] Arun DAS and Paul RAD. « Opportunities and Challenges in Explainable Artificial Intelligence (XAI): A Survey ». *CoRR*, abs/2006.11371, 2020.

[DRGM10] Emanuele DI ROSA, Enrico GIUNCHIGLIA, and Marco MARATEA. « Solving satisfiability problems with preferences ». *Constraints*, 15(4):485–515, 2010. 3.3.4

[DSZ16] Anupam DATTA, Shayak SEN, and Yair ZICK. « Algorithmic transparency via quantitative input influence: Theory and experiments with learning systems ». In *2016 IEEE symposium on security and privacy (SP)*, pages 598–617. IEEE, 2016. 1.1.2, 1.2.2

[DVK17] Finale DOSHI-VELEZ and Been KIM. « Towards a rigorous science of interpretable machine learning ». *arXiv preprint arXiv:1702.08608*, 2017. 1.1, 2.3, 5.2.2

[DVKB<sup>+</sup>17] Finale DOSHI-VELEZ, Mason KORTZ, Ryan BUDISH, Chris BAVITZ, Sam GERSHMAN, David O'BRIEN, Kate SCOTT, Stuart SCHIEBER, James WALDO, David WEINBERGER, and OTHERS. « Accountability of AI under the law: The role of explanation ». *arXiv preprint arXiv:1711.01134*, 2017. 1.1

[EMW97] Michael D ERNST, Todd D MILLSTEIN, and Daniel S WELD. « Automatic SAT-compilation of planning problems ». In *IJCAI*, volume 97, pages 1169–1176, 1997. 2.2.1

[ERLD17] Javid EBRAHIMI, Anyi RAO, Daniel LOWD, and Dejing DOU. « Hotflip: White-box adversarial examples for text classification ». *arXiv preprint arXiv:1712.06751*, 2017. 1.1.1

[EW01] André ELISSEEFF and Jason WESTON. « A kernel method for multi-labelled classification ». *Advances in neural information processing systems*, 14, 2001. 1.2

[FdHvE22] Hidde FOKKEMA, Rianne de HEIDE, and Tim van ERVEN. « Attribution-based Explanations that Provide Recourse Cannot be Robust ». *CoRR*, abs/2205.15834, 2022. 2.1
[FdSRGL22] Joao FERREIRA, Manuel de SOUSA RIBEIRO, Ricardo GONÇALVES, and Joao LEITE. « Looking Inside the Black-Box: Logic-based Explanations for Neural Networks ». In *Proceedings of the International Conference on Principles of Knowledge Representation and Reasoning*, volume 19, pages 432–442, 2022. 1.2, 1.3

[FH17] Nicholas FROSST and Geoffrey E. HINTON. « Distilling a Neural Network Into a Soft Decision Tree ». In Tarek R. BESOLD and Oliver KUTZ, editors, *Proceedings of the First International Workshop on Comprehensibility and Explanation in AI and ML 2017 co-located with 16th International Conference of the Italian Association for Artificial Intelligence (AI*IA 2017), Bari, Italy, November 16th and 17th, 2017*, volume 2071 of *CEUR Workshop Proceedings*. CEUR-WS.org, 2017. 1.2.2

[Fis36] Ronald A FISHER. « The use of multiple measurements in taxonomic problems ». *Annals of eugenics*, 7(2):179–188, 1936. (document), 1.5

[FJ18] Matteo FISCHETTI and Jason JO. « Deep neural networks and mixed integer linear optimization ». *Constraints*, 23(3):296–309, 2018.

[Fre14] Alex A FREITAS. « Comprehensible classification models: a position paper ». *ACM SIGKDD explorations newsletter*, 15(1):1–10, 2014. 1.2.1, 1.2.1

[Fri01] Jerome H FRIEDMAN. « Greedy function approximation: a gradient boosting machine ». *Annals of statistics*, pages 1189–1232, 2001. 1.2, 1.2.2, 1.3

[FSZ12] Alexander FELFERNIG, Monika SCHUBERT, and Christoph ZEHENTNER. « An efficient diagnosis algorithm for inconsistent constraint sets ». *AI EDAM*, 26(1):53–62, 2012.

[GA19] David GUNNING and David AHA. « DARPA’s explainable artificial intelligence (XAI) program ». *AI magazine*, 40(2):44–58, 2019. (document), 1.1, 1.1, 1.1.1, 1.2

[GBY+18] Leilani H GILPIN, David BAU, Ben Z YUAN, Ayesha BAJWA, Michael SPECTER, and Lalana KAGAL. « Explaining explanations: An overview of interpretability of machine learning ». In *2018 IEEE 5th International Conference on data science and advanced analytics (DSAA)*, pages 80–89. IEEE, 2018. 5.1

[GDL03] Muriel GEVREY, Ioannis DIMOPOULOS, and Sovan LEK. « Review and comparison of methods to study the contribution of variables in artificial neural network models ». *Ecological modelling*, 160(3):249–264, 2003. 1.2.2

[GIL18] Éric GRÉGOIRE, Yacine IZZA, and Jean-Marie LAGNIEZ. « Boosting MCSes Enumeration ». In *IJCAI*, pages 1309–1315, 2018. 3.3.4, 3.4.1, 4.5, 6.4.1

[GKBP15] Alex GOLDSTEIN, Adam KAPELNER, Justin BLEICH, and Emil PITKIN. « Peeking inside the black box: Visualizing statistical learning with plots of individual conditional expectation ». *journal of Computational and Graphical Statistics*, 24(1):44–65, 2015. (document), 1.2.2, 1.12

[GKC+18] Olivier GOUDET, Diviyan KALAINATHAN, Philippe CAILLOU, Isabelle GUYON, David LOPEZ-PAZ, and Michele SEBAG. Learning functional causal models with generative neural networks. In *Explainable and interpretable models in computer vision and machine learning*, pages 39–80. Springer, 2018. 1.2.2
[GLB<sup>+</sup>10] Michael L GATZA, Joseph E LUCAS, William T BARRY, Jong Wook KIM, Quanli WANG, Matthew D CRAWFORD, Michael B DATTO, Michael KELLEY, Bernard MATHEY-PREVOT, Anil POTTI, and OTHERS. « A pathway-based classification of human breast cancer ». *Proceedings of the National Academy of Sciences*, 107(15):6994–6999, 2010. 1.1

[GLM14] Éric GRÉGOIRE, Jean-Marie LAGNIEZ, and Bertrand MAZURE. « An experimentally efficient method for (MSS, CoMSS) partitioning ». In *Twenty-Eighth AAAI Conference on Artificial Intelligence*, 2014. 3.3.4

[GMP07] Eric GRÉGOIRE, Bertrand MAZURE, and Cédric PIETTE. « Boosting a Complete Technique to Find MSS and MUS Thanks to a Local Search Oracle. ». In *IJCAI-07*, volume 7, pages 2300–2305, 2007. 3.1, 3.3.1

[GMR<sup>+</sup>18] Riccardo GUIDOTTI, Anna MONREALE, Salvatore RUGGIERI, Franco TURINI, Fosca GIANNOTTI, and Dino PEDRESCHI. « A survey of methods for explaining black box models ». *ACM computing surveys (CSUR)*, 51(5):1–42, 2018.

[GMX<sup>+</sup>18] Wenbo GUO, Dongliang MU, Jun XU, Purui SU, Gang WANG, and Xinyu XING. « Lemna: Explaining deep learning based security applications ». In *proceedings of the 2018 ACM SIGSAC conference on computer and communications security*, pages 364–379, 2018. 1.2, 1.2.3, 1.3

[GPM<sup>+</sup>17] Kathrin GROSSE, Nicolas PAPERNOT, Praveen MANOHARAN, Michael BACKES, and Patrick MCDANIEL. « Adversarial examples for malware detection ». In *European symposium on research in computer security*, pages 62–79. Springer, 2017. 1.1.1

[GPSS19] Sachin GROVER, Chiara PULICE, Gerardo I SIMARI, and VS SUBRAHMANIAN. « Beef: Balanced english explanations of forecasts ». *IEEE Transactions on Computational Social Systems*, 6(2):350–364, 2019. 1.2, 1.3

[GR22] Niku GORJI and Sasha RUBIN. « Sufficient Reasons for Classifier Decisions in the Presence of Domain Constraints ». *AAAI*, February, 2022. 2.2

[GSS15] Ian J. GOODFELLOW, Jonathon SHLENS, and Christian SZEGEDY. « Explaining and Harnessing Adversarial Examples ». In Yoshua BENGIO and Yann LECUN, editors, *3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings*, 2015. 1.1.1, 1.1.2

[GT99] Fausto GIUNCHIGLIA and Paolo TRAVERSO. « Planning as model checking ». In *European Conference on Planning*, pages 1–20. Springer, 1999. 2.2.1

[Gup06] Anubhav GUPTA. « Learning abstractions for model checking ». PhD thesis, Carnegie Mellon University, 2006. 3.1, 3.3.4

[GWE<sup>+</sup>19] Yash GOYAL, Ziyan WU, Jan ERNST, Dhruv BATRA, Devi PARIKH, and Stefan LEE. « Counterfactual visual explanations ». In *International Conference on Machine Learning*, pages 2376–2384. PMLR, 2019. 1.2, 1.2.2

[Hau85] John HAUGELAND. « Artificial intelligence: the very idea », 1985. 1

[Hay94] Simon HAYKIN. *Neural networks: a comprehensive foundation*. Prentice Hall PTR, 1994. 1.1
[HCS<sup>+</sup>16] Itay HUBARA, Matthieu COURBARIAUX, Daniel SOUDRY, Ran EL-YANIV, and Yoshua BENGIO. « Binarized Neural Networks ». In *Advances in Neural Information Processing Systems*, volume 29, 2016. 3.4

[HD05] Jinbo HUANG and Adnan DARWICHE. « On compiling system models for faster and more scalable diagnosis ». In AAAI, pages 300–306, 2005. 2.2.1

[HDM<sup>+</sup>11] Johan HUYSMANS, Karel DEJAEGER, Christophe MUES, Jan VANTHIENEN, and Bart BAESENS. « An empirical evaluation of the comprehensibility of decision table, tree and rule based predictive models ». *Decision Support Systems*, 51(1):141–154, 2011. 1.2.1, 1.2.1

[HFS<sup>+</sup>18] Holger A HAENSSLE, Christine FINK, Roland SCHNEIDERBAUER, Ferdinand TOBERER, Timo BUHL, Andreas BLUM, A KALLOO, A Ben Hadj HASSEN, Luc THOMAS, A ENK, and OTHERS. « Man against machine: diagnostic performance of a deep learning convolutional neural network for dermoscopic melanoma recognition in comparison to 58 dermatologists ». *Annals of oncology*, 29(8):1836–1842, 2018.

[HH18] Satoshi HARA and Kohei HAYASHI. « Making Tree Ensembles Interpretable: A Bayesian Model Selection Approach ». In Amos J. STORKEY and Fernando PÉREZ-CRUZ, editors, *International Conference on Artificial Intelligence and Statistics, AIS-TATS 2018, 9-11 April 2018, Playa Blanca, Lanzarote, Canary Islands, Spain*, volume 84 of *Proceedings of Machine Learning Research*, pages 77–85. PMLR, 2018. 1.2.1

[HHDA18] Lisa Anne HENDRICKS, Ronghang HU, Trevor DARRELL, and Zeynep AKATA. « Generating Counterfactual Explanations with Natural Language ». *CoRR*, abs/1806.09809, 2018. 1.2, 1.2.2

[HIIM21] Xuanxiang HUANG, Yacine IZZA, Alexey IGNATIEV, and João MARQUES-SILVA. « On Efficiently Explaining Graph-Based Classifiers ». In Meghyn BIENVENU, Gerhard LAKEMEYER, and Esra ERDEM, editors, *Proceedings of the 18th International Conference on Principles of Knowledge Representation and Reasoning, KR 2021, Online event, November 3-12, 2021*, pages 356–367, 2021. 1.2.1, 2.2

[Hin19] Michael HIND. « Explaining explainable AI ». *XRDS: Crossroads, The ACM Magazine for Students*, 25(3):16–19, 2019. (document), 1.1.3, 1.3

[HKWL21] Shu HU, Lipeng KE, Xin WANG, and Siwei LYU. « TkML-AP: Adversarial Attacks to Top-k Multi-Label Learning ». In *Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)*, pages 7649–7657, October 2021.

[HM22] Xuanxiang HUANG and João MARQUES-SILVA. « On Deciding Feature Membership in Explanations of SDD & Related Classifiers ». *CoRR*, abs/2202.07553, 2022. 2.2

[Ho95] Tin Kam HO. « Random decision forests ». In *Proceedings of 3rd international conference on document analysis and recognition*, volume 1, pages 278–282. IEEE, 1995. 3.2.1

[HPS16] Moritz HARDT, Eric PRICE, and Nati SREBRO. « Equality of opportunity in supervised learning ». *Advances in neural information processing systems*, 29, 2016. 1.1.2
[HRS19]
Xiyang HU, Cynthia RUDIN, and Margo SELTZER. « Optimal sparse decision trees ». *Advances in Neural Information Processing Systems*, 32, 2019. 1.1

[HSJ+04]
Tarik HADZIC, Sathiamoorthy SUBBARAYAN, Rune M JENSEN, Henrik R ANDERSEN, Jesper MØLLER, and Henrik HULGAARD. « Fast backtrack-free product configuration using a precompiled solution space representation ». *small*, 10(1):3, 2004. 2.2.1

[HVD+15]
Geoffrey HINTON, Oriol VINYALS, Jeff DEAN, and OTHERS. « Distilling the knowledge in a neural network ». *arXiv preprint arXiv:1503.02531*, 2(7), 2015. 1.2.2

[HY01]
David J HAND and Keming YU. « Idiot's Bayes—not so stupid after all? ». *International statistical review*, 69(3):385-398, 2001. 1.1

[HYHI21]
Kazuaki HANAWA, Sho YOKOI, Satoshi HARA, and Kentaro INUI. « Evaluation of Similarity-based Explanations ». In *9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021*. OpenReview.net, 2021. 2.3

[Ign20]
Alexey IGNATIEV. « Towards Trustable Explainable AI ». In *IJCAI*, pages 5154-5158, 2020. 1.2.2, 2.1, 2.2, 2.2.2, 3.3.4, 5

[IIM20]
Yacine IZZA, Alexey IGNATIEV, and João MARQUES-SILVA. « On Explaining Decision Trees ». *CoRR*, abs/2010.11034, 2020. 1.2.2, 3.3.4

[IIM22]
Yacine IZZA, Alexey IGNATIEV, and João MARQUES-SILVA. « On Tackling Explanation Redundancy in Decision Trees ». *J. Artif. Intell. Res.*, 75:261-321, 2022. 2.2, 2.2.2

[IIN+21]
Yacine IZZA, Alexey IGNATIEV, Nina NARODYTSKA, Martin C. COOPER, and João MARQUES-SILVA. « Efficient Explanations With Relevant Sets ». *CoRR*, abs/2106.00546, 2021. 2.2

[IIN+22]
Yacine IZZA, Alexey IGNATIEV, Nina NARODYTSKA, Martin C COOPER, and Joao MARQUES-SILVA. « Provably Precise, Succinct and Efficient Explanations for Decision Trees ». *arXiv preprint arXiv:2205.09569*, 2022. 2.2

[IM21]
Yacine IZZA and João MARQUES-SILVA. « On Explaining Random Forests with SAT ». In Zhi-Hua ZHOU, editor, *Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI 2021, Virtual Event / Montreal, Canada, 19-27 August 2021*, pages 2584-2591. ijcai.org, 2021. 1.2.2, 3.3.4

[IMM18]
Alexey IGNATIEV, Antonio MORGADO, and João MARQUES-SILVA. « PySAT: A Python Toolkit for Prototyping with SAT Oracles ». In *SAT*, pages 428-437, 2018. 6.4.1

[IMMS18]
Alexey IGNATIEV, Antonio MORGADO, and João MARQUES-SILVA. « PySAT: A Python toolkit for prototyping with SAT oracles ». In *International Conference on Theory and Applications of Satisfiability Testing*, pages 428-437. Springer, 2018. 3.3.4

[IMS21]
Alexey IGNATIEV and João MARQUES-SILVA. « SAT-based rigorous explanations for decision lists ». In *International Conference on Theory and Applications of Satisfiability Testing*, pages 251-269. Springer, 2021. 1.2.2, 2.2, 2.2.2
[INAM20] Alexey IGNATIEV, Nina NARODYTSKA, Nicholas ASHER, and João MARQUES-SILVA. « On Relating 'Why?' and 'Why Not?' Explanations ». CoRR, abs/2012.11067, 2020. 1.2.2

[INAMS20] Alexey IGNATIEV, Nina NARODYTSKA, Nicholas ASHER, and João MARQUES-SILVA. « From contrastive to abductive explanations and back again ». In International Conference of the Italian Association for Artificial Intelligence, pages 335–355. Springer, 2020. 3, 3.3.3, 3.3.4

[INM19] Alexey IGNATIEV, Nina NARODYTSKA, and João MARQUES-SILVA. « On Validating, Repairing and Refining Heuristic ML Explanations ». CoRR, abs/1907.02509, 2019. 2.1

[INMS19a] Alexey IGNATIEV, Nina NARODYTSKA, and João MARQUES-SILVA. « Abduction-based explanations for machine learning models ». In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 1511–1519, 2019. 1.2, 1.2, 1.2.2, 1.2.2, 1.2.2, 1.3, 2.2, 2.2.2, 3, 5

[INMS19b] Alexey IGNATIEV, Nina NARODYTSKA, and João MARQUES-SILVA. « On Relating Explanations and Adversarial Examples ». In Advances in Neural Information Processing Systems, volume 32, 2019. 1.2.2, 2.2, 2.2.2, 5, 5.1.1

[IPLMS15] Alexey IGNATIEV, Alessandro PREVITI, Mark LIFFITON, and João MARQUES-SILVA. « Smallest MUS extraction with minimal hitting set dualization ». In International Conference on Principles and Practice of Constraint Programming, pages 173–182. Springer, 2015. 3.3.4

[IPNMS18] Alexey IGNATIEV, Filipe PEREIRA, Nina NARODYTSKA, and João MARQUES-SILVA. « A SAT-based approach to learn explainable decision sets ». In International Joint Conference on Automated Reasoning, pages 627–645. Springer, 2018. 1.1

[JBB$^{+21}$] Sérgio JESUS, Catarina BELÉM, Vladimir BALAYAN, João BENTO, Pedro SALEIRO, Pedro BIZARRO, and João GAMA. « How can I choose an explainer? An Application-grounded Evaluation of Post-hoc Explanations ». In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, pages 805–815, 2021. 2.3

[JCS$^{+20}$] Jongbin JUNG, Connor CONCANNON, Ravi SHROFF, Sharad GOEL, and Daniel G GOLDSTEIN. « Simple rules to guide expert classifications ». Journal of the Royal Statistical Society: Series A (Statistics in Society), 183(3):771–800, 2020. 1.2

[JG20] Alon JACOBI and Yoav GOLDBERG. « Towards Faithfully Interpretable NLP Systems: How Should We Define and Evaluate Faithfulness? ». In Dan JURAFSKY, Joyce CHAI, Natalie SCHLUTER, and Joel R. TETREAULT, editors, Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, pages 4198–4205. Association for Computational Linguistics, 2020. 2.3, 5.2.2

[JKN04] Ulf JOHANSSON, Rikard KÖNIG, and Lars NIKLASSON. « The truth is in there-rule extraction from opaque models using genetic programming. ». In FLAIRS Conference, pages 658–663. Miami Beach, FL, 2004. 1.2, 1.3

[JKV$^{+19}$] Shalmali JOSHI, Oluwasanmi KOYEJO, Warut VIJITBENJARONK, Been KIM, and Joy-deep GHOSH. « Towards Realistic Individual Recourse and Actionable Explanations in Black-Box Decision Making Systems ». CoRR, abs/1907.09615, 2019. 1.2.2
[JMD+05] Aleks JAKULIN, Martin MOŽINA, Janez DEMŠAR, Ivan BRATKO, and Blaž ZUPAN. « Nomograms for visualizing support vector machines ». In *Proceedings of the eleventh ACM SIGKDD international conference on Knowledge discovery in data mining*, pages 108–117, 2005. 1.2.2

[JSS15] Said JABBOUR, Lakhdar SAIS, and Yakoub SALHI. « Decomposition based SAT encodings for itemset mining problems ». In *Pacific-Asia Conference on Knowledge Discovery and Data Mining*, pages 662–674. Springer, 2015. 4.6

[Jun04] Ulrich JUNKER. « Preferred explanations and relaxations for over-constrained problems ». In AAAI-2004, 2004.

[JV00] Rune M JENSEN and Manuela M VELOSO. « OBDD-based universal planning for synchronized agents in non-deterministic domains ». *Journal of Artificial Intelligence Research*, 13:189–226, 2000. 2.2.1

[KAAS12] Toshihiro KAMISHIMA, Shotaro AKAHO, Hideki ASOH, and Jun SAKUMA. « Fairness-aware classifier with prejudice remover regularizer ». In *Joint European conference on machine learning and knowledge discovery in databases*, pages 35–50. Springer, 2012. 1.1.2

[Kas21] Atoosa KASIRZADEH. « Reasons, Values, Stakeholders: A Philosophical Framework for Explainable Artificial Intelligence ». In Madeleine Clare ELISH, William ISAAC, and Richard S. ZEMEL, editors, *FAccT ’21: 2021 ACM Conference on Fairness, Accountability, and Transparency, Virtual Event / Toronto, Canada, March 3-10, 2021*, Page 14. ACM, 2021. 1.1.2

[KBBV20] Amir-Hossein KARIMI, Gilles BARTHE, Borja BALLE, and Isabel VALERA. « Model-agnostic counterfactual explanations for consequential decisions ». In *International Conference on Artificial Intelligence and Statistics*, pages 895–905. PMLR, 2020. 1.2, 1.2.2, 1.2.2

[KBD+17] Guy KATZ, Clark BARRETT, David L DILL, Kyle JULIAN, and Mykel J KOCHENDERFER. « Reluplex: An efficient SMT solver for verifying deep neural networks ». In *International conference on computer aided verification*, pages 97–117. Springer, 2017. 2.2.2

[KGJS15] Been KIM, Elena GLASSMAN, Brittney JOHNSON, and Julie SHAH. « iBCM: Interactive Bayesian case model empowering humans via intuitive interaction ». 2015. 1.1, 1.3

[KK19a] Jakko KEMPER and Daan KOLKMAN. « Transparent to whom? No algorithmic accountability without a critical audience ». *Information, Communication & Society*, 22(14):2081–2096, 2019. 1.1.2

[KK19b] Eoin M KENNY and Mark T KEANE. « Twin-systems to explain artificial neural networks using case-based reasoning: Comparative tests of feature-weighting methods in ANN-CBR twins for XAI ». In *Twenty-Eighth International Joint Conferences on Artificial Intelligence (IJCAI), Macao, 10-16 August 2019*, pages 2708–2715, 2019. (document), 1.2.2, 1.14, 1.2.2
[KK21] Ioannis KAKOGEORGIOU and Konstantinos KARANTZALOS. « Evaluating explainable artificial intelligence methods for multi-label deep learning classification tasks in remote sensing ». *International Journal of Applied Earth Observation and Geoinformation*, 103:102520, 2021.

[KKK16] Been KIM, Rajiv KHANNA, and Oluwasanmi O KOYEJO. « Examples are not enough, learn to criticize! criticism for interpretability ». *Advances in neural information processing systems*, 29, 2016. 1.2, 1.2.2, 1.3

[KKS07] Ivo KONDAPANENI, Pavel KORDÍK, and Pavel SLAVÍK. « Visualization techniques utilizing the sensitivity analysis of models ». In *2007 Winter Simulation Conference*, pages 730–737. IEEE, 2007. 1.2, 1.2.2, 1.2.2, 1.3

[KRS14] Been KIM, Cynthia RUDIN, and Julie A SHAH. « The bayesian case model: A generative approach for case-based reasoning and prototype classification ». *Advances in neural information processing systems*, 27, 2014. 1.2.2

[KSB99] R KRISHNAN, G SIVAKUMAR, and P BHATTACHARYA. « Extracting decision trees from trained neural networks ». *Pattern recognition*, 32(12), 1999. 1.2.1

[KSH12] Alex KRIZHEVSKY, Ilya SUTSKEVER, and Geoffrey E HINTON. « Imagenet classification with deep convolutional neural networks ». *Advances in neural information processing systems*, 25, 2012. 1

[KTKA20] Kentaro KANAMORI, Takuya TAKAGI, Ken KOBAYASHI, and Hiroki ARIMURA. « DACE: Distribution-Aware Counterfactual Explanation by Mixed-Integer Linear Optimization ». In *IJCAI*, pages 2855–2862, 2020. 1.2, 1.2.2

[KWG+18] Been KIM, Martin WATTENBERG, Justin GILMER, Carrie CAI, James WEXLER, Fernanda VIEGAS, and OTHERS. « Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav) ». In *International conference on machine learning*, pages 2668–2677. PMLR, 2018. 1.2, 1.3

[LBBH98] Yann LE CUN, Léon BOTTOU, Yoshua BENGIO, and Patrick HAFFNER. « Gradient-based learning applied to document recognition ». *Proceedings of the IEEE*, 86(11):2278–2324, 1998. 33

[LBH15] Yann LE CUN, Yoshua BENGIO, and Geoffrey HINTON. « Deep learning ». *nature*, 521(7553):436–444, 2015. 1

[LBJ16] Tao LEI, Regina BARZILAY, and Tommi S. JAAKKOLA. « Rationalizing Neural Predictions ». In Jian SU, Xavier CARRERAS, and Kevin DUH, editors, *Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, EMNLP 2016*, Austin, Texas, USA, November 1-4, 2016, pages 107–117. The Association for Computational Linguistics, 2016. 1.2, 1.3, 2.3, 5.2.2

[LBL16] Himabindu LAKKARAJU, Stephen H BACH, and Jure LESKOVEC. « Interpretable decision sets: A joint framework for description and prediction ». In *Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining*, pages 1675–1684, 2016. 1.2.1, 14
[LC01] Stan LIPOVETSKY and Michael CONKLIN. « Analysis of regression in game theory approach ». *Applied Stochastic Models in Business and Industry*, 17(4):319–330, 2001. 1.2.2

[LCH<sup>+</sup>19] Isaac LAGE, Emily CHEN, Jeffrey HE, Menaka NARAYANAN, Been KIM, Sam GERSHMAN, and Finale DOSHI-VELEZ. « An evaluation of the human-interpretability of explanation ». *arXiv preprint arXiv:1902.00006*, 2019. 2.3, 5.2.2

[LCLS17] Yanpei LIU, Xinyun CHEN, Chang LIU, and Dawn SONG. « Delving into Transferable Adversarial Examples and Black-box Attacks ». In *5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings*. OpenReview.net, 2017. 1.1.2

[LDB<sup>+</sup>12] Oscar LUACES, Jorge DÍEZ, José BARRANQUERO, Juan José del COZ, and Antonio BAHAMONDE. « Binary relevance efficacy for multilabel classification ». *Progress in Artificial Intelligence*, 1(4):303–313, 2012. 1

[LEC<sup>+</sup>20a] Scott M LUNDBERG, Gabriel ERION, Hugh CHEN, Alex DEGRAVE, Jordan M PRUTKIN, Bala NAIR, Ronit KATZ, Jonathan HIMMELFARB, Nisha BANSAL, and Su-In LEE. « From local explanations to global understanding with explainable AI for trees ». *Nature machine intelligence*, 2(1):56–67, 2020. 1.2, 1.3

[LEC<sup>+</sup>20b] Scott M. LUNDBERG, Gabriel G. ERION, Hugh CHEN, Alex J. DEGRAVE, Jordan M. PRUTKIN, Bala NAIR, Ronit KATZ, Jonathan HIMMELFARB, Nisha BANSAL, and Su-In LEE. « From local explanations to global understanding with explainable AI for trees ». *Nat. Mach. Intell.*, 2(1):56–67, 2020. 1.2.2

[LEL18] Scott M LUNDBERG, Gabriel G ERION, and Su-In LEE. « Consistent individualized feature attribution for tree ensembles ». *arXiv preprint arXiv:1802.03888*, 2018. 1.2.2

[Lib05] Paolo LIBERATORE. « Redundancy in logic I: CNF propositional formulae ». *Artificial Intelligence*, 163(2):203–232, 2005. 3.1, 3.3.4

[Lip18] Zachary C LIPTON. « The Mythos of Model Interpretability: In machine learning, the concept of interpretability is both important and slippery. ». *Queue*, 16(3):31–57, 2018. 1, 1.1, 1.2.2

[LK21] Arnaud Van LOOVEREN and Janis KLAISE. « Interpretable counterfactual explanations guided by prototypes ». In *Joint European Conference on Machine Learning and Knowledge Discovery in Databases*, pages 650–665. Springer, 2021. 1.2, 1.2.2

[LKB<sup>+</sup>17] Geert LITJENS, Thijs KOOI, Babak Ehteshami BEJNORDI, Arnaud Arindra Adiyoso SETIO, Francesco CIOMPI, Mohsen GHAFOORIAN, Jeroen Awm VAN DER LAAK, Bram VAN GINNEKEN, and Clara I SÁNCHEZ. « A survey on deep learning in medical image analysis ». *Medical image analysis*, 42:60–88, 2017. 1.1

[LKCL19] Himabindu LAKKARAJU, Ece KAMAR, Rich CARUANA, and Jure LESKOVEC. « Faithful and customizable explanations of black box models ». In *Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society*, pages 131–138, 2019. 1.2.2, 2.3

[LKLH19] Shusen LIU, Bhavya KAILKHURA, Donald LOVELAND, and Yong HAN. « Generative counterfactual introspection for explainable deep learning ». In *2019 IEEE Global*
<table>
  <tr>
    <td>Conference on Signal and Information Processing (GlobalSIP), pages 1–5. IEEE, 2019.<br>1.2.2</td>
  </tr>
  <tr>
    <td>[LL17] Scott M LUNDBERG and Su-In LEE. « A unified approach to interpreting model predictions ». <i>Advances in neural information processing systems</i>, 30, 2017. 1.2, 1.2, 1.2.2, 1.2.2, 1.2.2, 1.2.3, 1.3, 2.1, 3, 33, 5, 5.1.1, 5.3, 6.1, 6.5</td>
  </tr>
  <tr>
    <td>[LLCR18] Oscar Li, Hao LIU, Chaofan CHEN, and Cynthia RUDIN. « Deep learning for case-based reasoning through prototypes: A neural network that explains its predictions ». In <i>Proceedings of the AAAI Conference on Artificial Intelligence</i>, volume 32, 2018. 1.2.2</td>
  </tr>
  <tr>
    <td>[LLM<sup>+</sup>19] Thibault LAUGEL, Marie-Jeanne LESOT, Christophe MARSALA, X. RENARD, and Marcin DETYNIECKI. « The Dangers of Post-hoc Interpretability: Unjustified Counterfactual Explanations ». In <i>IJCAI</i>, 2019. 1.2.2</td>
  </tr>
  <tr>
    <td>[LLS<sup>+</sup>17] Bin LIANG, Hongcheng LI, Miaoqiang SU, Pan BIAN, Xirong LI, and Wenchang SHI. « Deep text classification can be fooled ». <i>arXiv preprint arXiv:1704.08006</i>, 2017. 1.1.2</td>
  </tr>
  <tr>
    <td>[LNPT18] Francesco LEOFANTE, Nina NARODYTSKA, Luca PULINA, and Armando TACCHELLA. « Automated verification of neural networks: Advances, challenges and perspectives ». <i>arXiv preprint arXiv:1805.09938</i>, 2018. 2.2.2</td>
  </tr>
  <tr>
    <td>[LO03] Tao Li and Mitsunori OGIHARA. « Detecting emotion in music ». 2003. 1.2</td>
  </tr>
  <tr>
    <td>[LOL<sup>+</sup>18] Bruno LEPRI, Nuria OLIVER, Emmanuel LETOUZÉ, Alex PENTLAND, and Patrick VINCK. « Fair, transparent, and accountable algorithmic decision-making processes ». <i>Philosophy & Technology</i>, 31(4):611–627, 2018. 1.1, 1.1.2</td>
  </tr>
  <tr>
    <td>[LPK20] Pantelis LINARDATOS, Vasilis PAPASTEFANOPOULOS, and Sotiris KOTSIANTIS. « Explainable AI: A review of machine learning interpretability methods ». <i>Entropy</i>, 23(1):18, 2020. 1.2.3, 5.1.1</td>
  </tr>
  <tr>
    <td>[LPK21] Pantelis LINARDATOS, Vasilis PAPASTEFANOPOULOS, and Sotiris KOTSIANTIS. « Explainable AI: A Review of Machine Learning Interpretability Methods ». <i>Entropy</i>, 23(1), 2021. 5.1.1, 6.2.1</td>
  </tr>
  <tr>
    <td>[LPMMS16] Mark H LIFFITON, Alessandro PREVITI, Ammar MALIK, and Joao MARQUES-SILVA. « Fast, flexible MUS enumeration ». <i>Constraints</i>, 21(2):223–250, 2016. 3.1, 3.3.1</td>
  </tr>
  <tr>
    <td>[LPNC<sup>+</sup>17] David LOPEZ-PAZ, Robert NISHIHARA, Soumith CHINTALA, Bernhard SCHOLKOPF, and Léon BOTTOU. « Discovering causal signals in images ». In <i>Proceedings of the IEEE conference on computer vision and pattern recognition</i>, pages 6979–6987, 2017. 1.2.2</td>
  </tr>
  <tr>
    <td>[LR76] Hyafil LAURENT and Ronald L RIVEST. « Constructing optimal binary decision trees is NP-complete ». <i>Information processing letters</i>, 5(1):15–17, 1976. 1.2.1</td>
  </tr>
  <tr>
    <td>[LRBM08] Marie-Jeanne LESOT, Maria RIFQI, and Bernadette BOUCHON-MEUNIER. Fuzzy prototypes: From a cognitive view to a machine learning principle. In <i>Fuzzy Sets and Their Extensions: Representation, Aggregation and Models</i>, pages 431–452. Springer, 2008. 1.2.2</td>
  </tr>
</table>
[LRL<sup>+</sup>18] Thibault LAUGEL, X. RENARD, Marie-Jeanne LESOT, Christophe MARSALA, and Marcin DETYNIECKI. « Defining Locality for Surrogates in Post-hoc Interpretability. » *ArXiv*, abs/1806.07498, 2018. 1.2.2

[LRMM15] Benjamin LETHAM, Cynthia RUDIN, Tyler H MCCORMICK, and David MADIGAN. « Interpretable classifiers using rules and bayesian analysis: Building a better stroke prediction model ». *The Annals of Applied Statistics*, 9(3):1350–1371, 2015. 1.1, 1.3

[LS08] Mark H LIFFITON and Karem A SAKALLAH. « Algorithms for computing minimal unsatisfiable subsets of constraints ». *Journal of Automated Reasoning*, 40(1):1–33, 2008. 3.1, 3.3.1, 3.3.4, 3.3.4, 3.3.4, 5.2, 6.6

[LS09] Mark H LIFFITON and Karem A SAKALLAH. « Generalizing core-guided Max-SAT ». In *International Conference on Theory and Applications of Satisfiability Testing*, pages 481–494. Springer, 2009. 3.3.4

[Mar91] Pierre MARQUIS. « Extending abduction from propositional to first-order logic ». In *International Workshop on Fundamentals of Artificial Intelligence Research*, pages 141–155. Springer, 1991. 2.2.2

[MB18] D Douglas MILLER and Eric W BROWN. « Artificial intelligence in medical practice: the question to the answer? ». *The American journal of medicine*, 131(2):129–133, 2018. 1.1

[MCS<sup>+</sup>21] Stefano MELACCI, Gabriele CIRAVEGNA, Angelo SOTGIU, Ambra DEMONTIS, Battista BIGGIO, Marco GORI, and Fabio ROLI. « Domain Knowledge Alleviates Adversarial Attacks in Multi-Label Classifiers ». *IEEE Transactions on Pattern Analysis and Machine Intelligence*, pages 1–1, 2021.

[MCV<sup>+</sup>18] Rory MCGRATH, Luca COSTABELLO, Chan Le VAN, Paul SWEENEY, Farbod KAMIAB, Zhao SHEN, and Freddy LÉCUÉ. « Interpretable Credit Application Predictions With Counterfactual Explanations ». *CoRR*, abs/1811.05245, 2018. 3.3.3

[MDFF16] Seyed-Mohsen MOOSAVI-DEZFOOLI, Alhussein FAWZI, and Pascal FROSSARD. « Deepfool: a simple and accurate method to fool deep neural networks ». In *Proceedings of the IEEE conference on computer vision and pattern recognition*, pages 2574–2582, 2016. 1, 1.1.2

[MDFFF17] Seyed-Mohsen MOOSAVI-DEZFOOLI, Alhussein FAWZI, Omar FAWZI, and Pascal FROSSARD. « Universal adversarial perturbations ». In *Proceedings of the IEEE conference on computer vision and pattern recognition*, pages 1765–1773, 2017. 1.1.2

[Mel21] Marco MELIS. « Explaining Vulnerability of Machine Learning to Adversarial Attacks ». 2021. (document), 1.4

[MG20] Samaneh MAHDAVIFAR and Ali A GHORBANI. « DeNNeS: deep embedded neural network expert system for detecting cyber attacks ». *Neural Computing and Applications*, 32(18):14753–14780, 2020. 1.2, 1.3

[Mil19] Tim MILLER. « Explanation in artificial intelligence: Insights from the social sciences ». *Artificial intelligence*, 267:1–38, 2019. 1, 1.1, 1.1.1
[Mil21] Tim MILLER. « Contrastive explanation: A structural-model approach ». *The Knowledge Engineering Review*, 36, 2021. 1.2.2

[MIPMS16] Carlos MENCÍA, Alexey IGNATIEV, Alessandro PREVITI, and João MARQUES-SILVA. « MCS extraction with sublinear oracle queries ». In *International Conference on Theory and Applications of Satisfiability Testing*, pages 342–360. Springer, 2016. 3.1, 3.3.1, 3.3.4

[MLA+05] Maher MNEIMNEH, Inês LYNCE, Zaher ANDRAUS, João MARQUES-SILVA, and Karem SAKALLAH. « A branch-and-bound algorithm for extracting smallest minimal unsatisfiable formulas ». In *International Conference on Theory and Applications of Satisfiability Testing*, pages 467–474. Springer, 2005. 3.3.4

[MLB+17] Grégoire MONTAVON, Sebastian LAPUSCHKIN, Alexander BINDER, Wojciech SAMEK, and Klaus-Robert MÜLLER. « Explaining nonlinear classification decisions with deep taylor decomposition ». *Pattern recognition*, 65:211–222, 2017. 1.2.2, 1.2.3

[Mol22] Christoph MOLNAR. *Interpretable Machine Learning*. 2 edition, 2022. 1, 1.2, 1.2.1, 1.2.2, 1.2.2, 1.2.2

[MPMS15] Carlos MENCÍA, Alessandro PREVITI, and João MARQUES-SILVA. « Literal-based MCS extraction ». In *Twenty-Fourth International Joint Conference on Artificial Intelligence*, 2015. 3.1, 3.3.1, 3.3.4, 3.3.4

[MSGC+20] João MARQUES-SILVA, Thomas GERSPACHER, Martin COOPER, Alexey IGNATIEV, and Nina NARODYTSKA. « Explaining naive bayes and other linear classifiers with polynomial time and delay ». *Advances in Neural Information Processing Systems*, 33:20590–20600, 2020. 2.2, 3.3.4

[MSHJ+13] João MARQUES-SILVA, Federico HERAS, Mikolás JANOTA, Alessandro PREVITI, and Anton BELOV. « On computing minimal correction subsets ». In *Twenty-Third International Joint Conference on Artificial Intelligence*. Citeseer, 2013. 3.3.4, 3.3.4

[MST20] Ramaravind K MOTHILAL, Amit SHARMA, and Chenhao TAN. « Explaining machine learning classifiers through diverse counterfactual explanations ». In *Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency*, pages 607–617, 2020. 1.2, 1.2.2, 1.2.2, 3.3.3

[MT07] D Michael MILLER and Mitchell A THORNTON. « Multiple valued logic: Concepts and representations ». *Synthesis lectures on digital circuits and systems*, 2(1):1–127, 2007.

[MTR16] Sameer Singh MARCO TULIO RIBEIRO. « Local interpretable model-agnostic explanations (LIME): An introduction ». Aug 2016. (document), 1.10

[MTS19] Divyat MAHAJAN, Chenhao TAN, and Amit SHARMA. « Preserving causal constraints in counterfactual explanations for machine learning classifiers ». *arXiv preprint arXiv:1912.03277*, 2019. 1.2, 1.2.2, 1.2.2

[MVED17] Dmitry M MALIOUTOV, Kush R VARSHNEY, Amin EMAD, and Sanjeeb DASH. Learning interpretable classification rules with boolean compressed sensing. In *Transparent Data Mining for Big and Small Data*, pages 95–121. Springer, 2017. 1.2.1
[MWM18] Daniel L. MARINO, Chathurika S. WICKRAMASINGHE, and Milos MANIC. « An Adversarial Approach for Explainable AI in Intrusion Detection Systems ». In *IECON 2018 - 44th Annual Conference of the IEEE Industrial Electronics Society*, pages 3237–3243, 2018. 1.2, 1.3

[MZR21] Sina MOHSENI, Niloofar ZAREI, and Eric D RAGAN. « A multidisciplinary survey and framework for design and evaluation of explainable AI systems ». *ACM Transactions on Interactive Intelligent Systems (TiiS)*, 11(3-4):1–45, 2021. 2.3

[NAC02] Haydemar NÚÑEZ, Cecilio ANGULO, and Andreu CATALÀ. « Rule extraction from support vector machines. ». In *Esann*, pages 107–112, 2002. 1.2.1

[NBMS18] Nina NARODYTSKA, Nikolaj BJØRNER, Maria Cristina MARINESCU, and Mooly SAGIV. « Core-guided minimal correction set and core enumeration ». In *IJCAI International Joint Conference on Artificial Intelligence: Stockholm, 13-19 July 2018*, pages 1353–1361. IJCAI, 2018. 3.1, 3.3.1, 3.3.4

[NDY$^{+}$16] Anh NGUYEN, Alexey DOSOVITSKIY, Jason YOSINSKI, Thomas BROX, and Jeff CLUNE. « Synthesizing the preferred inputs for neurons in neural networks via deep generator networks ». *Advances in neural information processing systems*, 29, 2016. 1.2.2

[NGC$^{+}$20] Woo-Jeoung NAM, Shir GUR, Jaesik CHOI, Lior WOLF, and Seong-Whan LEE. « Relative attributing propagation: Interpreting the comparative contributions of individual units in deep neural networks ». In *Proceedings of the AAAI Conference on Artificial Intelligence*, volume 34, pages 2501–2508, 2020.

[NHWF21] Anna NGUYEN, Daniel HAGENMAYER, Tobias WELLER, and Michael FÄRBER. « Explaining Convolutional Neural Networks by Tagging Filters », 2021. 5.1.1, 6.1

[Nig16] Priyanka NIGAM. « Applying deep learning to ICD-9 multi-label classification from medical records ». Technical report, Technical report, Stanford University, 2016. 4.2

[NK17] Nina NARODYTSKA and Shiva Prasad KASIVISWANATHAN. « Simple Black-Box Adversarial Attacks on Deep Neural Networks. ». In *CVPR Workshops*, volume 2, Page 2, 2017. 1.1.2

[NKR$^{+}$18] Nina NARODYTSKA, Shiva KASIVISWANATHAN, Leonid RYZHYK, Mooly SAGIV, and Toby WALSH. « Verifying properties of binarized deep neural networks ». In *Proceedings of the AAAI Conference on Artificial Intelligence*, volume 32, 2018. 2.2.1, 2.2.2, 3.2.1, 4.4.1

[NRS13] Alexander NADEL, Vadim RYVCHIN, and Ofer STRICHMAN. « Efficient MUS extraction with resolution ». In *2013 Formal Methods in Computer-Aided Design*, pages 197–200. IEEE, 2013. 3.3.4

[NRS14] Alexander NADEL, Vadim RYVCHIN, and Ofer STRICHMAN. « Accelerated deletion-based extraction of minimal unsatisfiable cores ». *Journal on Satisfiability, Boolean Modeling and Computation*, 9(1):27–51, 2014. 3.3.4

[NYC15] Anh NGUYEN, Jason YOSINSKI, and Jeff CLUNE. « Deep neural networks are easily fooled: High confidence predictions for unrecognizable images ». In *Proceedings of the IEEE conference on computer vision and pattern recognition*, pages 427–436, 2015. 1
[NZ17] Robin NIX and Jian ZHANG. « Classification of Android apps and malware using deep neural networks ». In *2017 International joint conference on neural networks (IJCNN)*, pages 1871–1878. IEEE, 2017. 1.1

[OD15] Umut OZTOK and Adnan DARWICHE. « A top-down compiler for sentential decision diagrams ». In *Twenty-Fourth International Joint Conference on Artificial Intelligence*, 2015. 2.2.1

[OMS17] Chris OLAH, Alexander MORDVINTSEV, and Ludwig SCHUBERT. « Feature visualization ». *Distill*, 2(11):e7, 2017. 1.2.2

[OSC07] Olga OHRIMENKO, Peter J STUCKEY, and Michael CODISH. « Propagation= lazy clause generation ». In *International Conference on Principles and Practice of Constraint Programming*, pages 544–558. Springer, 2007. 3.1

[PGA08] Kemal POLAT, Salih GÜNEŞ, and Ahmet ARSLAN. « A cascade learning system for classification of diabetes disease: Generalized discriminant analysis and least square support vector machine ». *Expert systems with applications*, 34(1):482–487, 2008. 1.1

[PGG$^{+}$19] Dino PEDRESCHI, Fosca GIANNOTTI, Riccardo GUIDOTTI, Anna MONREALE, Salvatore RUGGIERI, and Franco TURINI. « Meaningful explanations of black box AI decision systems ». In *Proceedings of the AAAI conference on artificial intelligence*, volume 33, pages 9780–9784, 2019. 1.2.1

[PGMP19] Cecilia PANIGUTTI, Riccardo GUIDOTTI, Anna MONREALE, and Dino PEDRESCHI. « Explaining multi-label black-box classifiers for health applications ». In *International Workshop on Health Intelligence*, pages 97–110. Springer, 2019. 4.1, 6.1

[PHL$^{+}$17] Timo PEKKALA, Anette HALL, Jyrki LÖTJÖNEN, Jussi MATTILA, Hilkka SOININEN, Tiia NGANDU, Tiina LAATIKAINEN, Miia KIVIPELTO, and Alina SOLOMON. « Development of a late-life dementia prediction index with supervised machine learning in the population-based CAIDE study ». *Journal of Alzheimer's Disease*, 55(3):1055–1067, 2017. 1.1

[PLPN19] Badri N PATRO, Mayank LUNAYACH, Shivansh PATEL, and Vinay P NAMBOODIRI. « U-cam: Visual explanation using uncertainty based class activation maps ». In *Proceedings of the IEEE/CVF International Conference on Computer Vision*, pages 7444–7453, 2019. 5.1.1

[PMJ$^{+}$16] Nicolas PAPERNOT, Patrick MCDANIEL, Somesh JHA, Matt FREDRIKSON, Z Berkay CELIK, and Ananthram SWAMI. « The limitations of deep learning in adversarial settings ». In *2016 IEEE European symposium on security and privacy (EuroS&P)*, pages 372–387. IEEE, 2016. 1.1.1

[PMJMS17] Alessandro PREVITI, Carlos MENCÍA, Matti JÄRVISALO, and Joao MARQUES-SILVA. « Improving MCS enumeration via caching ». In *International Conference on Theory and Applications of Satisfiability Testing*, pages 184–194. Springer, 2017. 3.3.4

[PMJMS18] Alessandro PREVITI, Carlos MENCÍA, Matti JÄRVISALO, and Joao MARQUES-SILVA. « Premise set caching for enumerating minimal correction subsets ». In *Proceedings of the AAAI Conference on Artificial Intelligence*, volume 32, 2018. 3.1, 3.3.1, 3.3.4
[Qui87a] J Ross QUINLAN. « Generating production rules from decision trees. ». In *ijcai*, volume 87, pages 304–307. Citeseer, 1987.

[Qui87b] J. Ross QUINLAN. « Simplifying decision trees ». *International journal of man-machine studies*, 27(3):221–234, 1987. 1.2.1

[Rei87] Raymond REITER. « A theory of diagnosis from first principles ». *Artificial intelligence*, 32(1):57–95, 1987. 3.1, 2.2, 2.2.2, 3.3.4, 5.1.1

[RPF<sup>+</sup>21] Thomas ROJAT, Raphaël PUGET, David FILLIAT, Javier DEL SER, Rodolphe GELIN, and Natalia DÍAZ-RODRÍGUEZ. « Explainable artificial intelligence (xai) on timeseries data: A survey ». *arXiv preprint arXiv:2104.00950*, 2021. 1.2.2

[RPH08] Jesse READ, Bernhard PFAHRINGER, and Geoff HOLMES. « Multi-label classification using ensembles of pruned sets ». In *2008 eighth IEEE international conference on data mining*, pages 995–1000. IEEE, 2008. 1

[RPHF09] Jesse READ, Bernhard PFAHRINGER, Geoff HOLMES, and Eibe FRANK. « Classifier chains for multi-label classification ». In *Joint European conference on machine learning and knowledge discovery in databases*, pages 254–269. Springer, 2009. 1, 3

[RPHF11] Jesse READ, Bernhard PFAHRINGER, Geoff HOLMES, and Eibe FRANK. « Classifier chains for multi-label classification ». *Machine learning*, 85(3):333–359, 2011. 3

[RPR<sup>+</sup>10] Cynthia RUDIN, Rebecca J PASSONNEAU, Axinia RADEVA, Haimonti DUTTA, Steve IEROME, and Delfina ISAAC. « A process for predicting manhole events in Manhattan ». *Machine Learning*, 80(1):1–31, 2010. 1.1

[RRK90] Dennis W RUCK, Steven K ROGERS, and Matthew KABRISKY. « Feature selection using a multilayer perceptron ». *Journal of Neural Network Computing*, 2(2):40–48, 1990. 1.2.2

[RSE16] Loukmen REGAINIA, Sébastien SALVA, and Cedric ECUHCURS. « A classification methodology for security patterns to help fix software weaknesses ». In *2016 IEEE/ACS 13th International Conference of Computer Systems and Applications (AICCSA)*, pages 1–8. IEEE, 2016. 1.1

[RSG16] Marco Tulio RIBEIRO, Sameer SINGH, and Carlos GUESTRIN. « Why should I trust you? » Explaining the predictions of any classifier ». In *Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining*, pages 1135–1144, 2016. (document), 1.2, 1.2.1, 1.2.1, 1.2, 1.2.2, 1.2.2, 1.9, 1.2.3, 1.3, 2.1, 3, 3.2.2, 5, 5.1.1, 6.1, 6.5

[RSG18] Marco Tulio RIBEIRO, Sameer SINGH, and Carlos GUESTRIN. « Anchors: High-precision model-agnostic explanations ». In *Proceedings of the AAAI conference on artificial intelligence*, volume 32, 2018. 1.2, 1.2, 1.2.2, 1.2.2, 1.3, 2.1, 3.2.2, 5

[RU18] Cynthia RUDIN and Berk USTUN. « Optimized scoring systems: Toward trust in machine learning for healthcare and criminal justice ». *Interfaces*, 48(5):449–466, 2018. 1.1
[Rud19] Cynthia RUDIN. « Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead ». *Nature Machine Intelligence*, 1(5):206-215, 2019. 1.1

[Rus19] Chris RUSSELL. « Efficient search for diverse coherent explanations ». In *Proceedings of the Conference on Fairness, Accountability, and Transparency*, pages 20–28, 2019. 1.2, 1.2.2, 1.2.2

[Rym94] Ron RYMON. « An se-tree-based prime implicant generation algorithm ». *Annals of Mathematics and Artificial Intelligence*, 11(1):351–365, 1994. 2.2.2, 5.1.1

[SB75] Edward H SHORTLIFFE and Bruce G BUCHANAN. « A model of inexact reasoning in medicine ». *Mathematical biosciences*, 23(3-4):351–379, 1975. 1.1.1

[SB21] Kushal SINGLA and Subham BISWAS. « Machine learning explainability method for the multi-label classification model ». In *2021 IEEE 15th International Conference on Semantic Computing (ICSC)*, pages 337–340. IEEE, 2021. 4.1, 6.1

[SCD$^{+}$17] Ramprasaath R SELVARAJU, Michael COGSWELL, Abhishek DAS, Ramakrishna VEDANTAM, Devi PARIKH, and Dhruv BATRA. « Grad-cam: Visual explanations from deep networks via gradient-based localization ». In *Proceedings of the IEEE international conference on computer vision*, pages 618–626, 2017. (document), 1.2.2, 1.8, 1.2.3, 5.1.1

[SCD18a] Andy SHIH, Arthur CHOI, and Adnan DARWICHE. « Formal verification of Bayesian network classifiers ». In *International Conference on Probabilistic Graphical Models*, pages 427–438. PMLR, 2018. 2.2.1

[SCD18b] Andy SHIH, Arthur CHOI, and Adnan DARWICHE. « A symbolic approach to explaining bayesian network classifiers ». *arXiv preprint arXiv:1805.03364*, 2018. 1.2, 1.2, 1.2.2, 1.3, 2.2, 2.2.1, 2.2.2, 8, 3, 3.2.1, 3.2.1, 3.2.1, 5, 5.1.1

[SCD19] Andy SHIH, Arthur CHOI, and Adnan DARWICHE. « Compiling Bayesian Network Classifiers into Decision Graphs ». In *Proceedings of the AAAI-19*, volume 33, pages 7966–7974, 2019. 2.2, 2.2.1, 3, 3.2.1, 3.2.1, 4.4.1, 6.6

[SDBR14] Jost Tobias SPRINGENBERG, Alexey DOSOVITSKIY, Thomas BROX, and Martin RIEDMILLER. « Striving for simplicity: The all convolutional net ». *arXiv preprint arXiv:1412.6806*, 2014. (document), 1.2, 1.2.2, 1.8, 1.2.3, 1.3

[SDBR15] JT SPRINGENBERG, A DOSOVITSKIY, T BROX, and M RIEDMILLER. « Striving for simplicity: The all convolutional net. In arxiv: cs ». *arXiv preprint arXiv:1412.6806*, 2015. 5.1.1

[SDC19] Andy SHIH, Adnan DARWICHE, and Arthur CHOI. « Verifying binarized neural networks by Angluin-style learning ». In *International Conference on Theory and Applications of Satisfiability Testing*, pages 354–370. Springer, 2019. 2.2.2

[SGK17] Avanti SHRIKUMAR, Peyton GREENSIDE, and Anshul KUNDAJE. « Learning important features through propagating activation differences ». In *International conference on machine learning*, pages 3145–3153. PMLR, 2017. 1.2.2, 5.1.1
[SGM+21] Mattia SETZU, Riccardo GUIDOTTI, Anna MONREALE, Franco TURINI, Dino PEDRESCHI, and Fosca GIANNOTTI. « Glocalx-from local to global explanations of black box AI models ». *Artificial Intelligence*, 294:103457, 2021. 1.2, 1.3

[SGZS21] Maximilian SCHLEICH, Zixuan GENG, Yihong ZHANG, and Dan SUCIU. « GeCo: quality counterfactual explanations in real time ». *arXiv preprint arXiv:2101.01292*, 2021. (document), 1.13

[SH+08] Sajjad SIDDIQI, Jinbo HUANG, and OTHERS. « Probabilistic sequential diagnosis by compilation ». 2008. 2.2.1

[Sha53] LS SHAPLEY. « QUOTA SOLUTIONS OP n-PERSON GAMES1 ». *Edited by Emil Artin and Marston Morse*, Page 343, 1953. 1.2.2, 2.1

[Sha89] Murray SHANAHAN. « Prediction is Deduction but Explanation is Abduction. ». In *IJCAI*, volume 89, pages 1055–1060, 1989. 2.2.2

[SHG19] Shubham SHARMA, Jette HENDERSON, and Joydeep GHOSH. « Certifai: Counterfactual explanations for robustness, transparency, interpretability, and fairness of artificial intelligence models ». *arXiv preprint arXiv:1905.07857*, 2019. 1.2.2

[SHJ+20] Dylan SLACK, Sophie HILGARD, Emily JIA, Sameer SINGH, and Himabindu LAKKARAJU. « Fooling lime and shap: Adversarial attacks on post hoc explanation methods ». In *Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society*, pages 180–186, 2020. 1.2.2, 2.1

[Sin05] Carsten SINZ. « Towards an optimal CNF encoding of boolean cardinality constraints ». In *International conference on principles and practice of constraint programming*, pages 827–831. Springer, 2005. 3.2.1, 3.2.1

[ŠK14] Erik ŠTRUMBELJ and Igor KONONENKO. « Explaining prediction models and individual predictions with feature contributions ». *Knowledge and information systems*, 41(3):647–665, 2014. 1.2.2

[SKG09] Yang SONG, Aleksander KOŁCZ, and C Lee GILES. « Better Naive Bayes classification for high-precision spam detection ». *Software: Practice and Experience*, 39(11):1003–1024, 2009. 1.1

[SLZS22] Weidi SUN, Yuteng LU, Xiyue ZHANG, and Meng SUN. « DeepGlobal: A framework for global robustness verification of feedforward neural networks ». *Journal of Systems Architecture*, Page 102582, 2022. 1.2, 1.3

[SMA+18] Keiichi SHIMA, Daisuke MIYAMOTO, Hiroshi ABE, Tomohiro ISHIHARA, Kazuya OKADA, Yuji SEKIYA, Hirochika ASAI, and Yusuke DOI. « Classification of URL bitstreams using bag of bytes ». In *2018 21st Conference on Innovation in Clouds, Internet and Networks and Workshops (ICIN)*, pages 1–5. IEEE, 2018. 1.1

[SNPS18] Nathan SHONE, Tran Nguyen NGOC, Vu Dinh PHAI, and Qi SHI. « A deep learning approach to network intrusion detection ». *IEEE transactions on emerging topics in computational intelligence*, 2(1):41–50, 2018. 1.1
[SPG<sup>+</sup>19] Shriansh SRIVASTAVA, J PRIYADARSHINI, Sachin GOPAL, Sanchay GUPTA, and Har Shobhit DAYAL. Optical character recognition on bank cheques using 2D convolution neural network. In *Applications of Artificial Intelligence Techniques in Engineering*, pages 589–596. Springer, 2019. 1.1

[SRA<sup>+</sup>08] Andrea SALTELLI, Marco RATTO, Terry ANDRES, Francesca CAMPOLONGO, Jessica CARIBONI, Debora GATELLI, Michaela SAISANA, and Stefano TARANTOLA. *Global sensitivity analysis: the primer*. John Wiley & Sons, 2008. 1.2, 1.3

[SSDC20] Weijia SHI, Andy SHIH, Adnan DARWICHE, and Arthur CHOI. « On Tractable Representations of Binary Neural Networks ». In Diego CALVANESE, Esra ERDEM, and Michael THIELSCHER, editors, *Proceedings of the 17th International Conference on Principles of Knowledge Representation and Reasoning, KR 2020, Rhodes, Greece, September 12-18, 2020*, pages 882–892, 2020. 2.2, 2.2.1, 3.3.4, 4.4.1, 6.6

[SSF14] Paulo SHAKARIAN, Gerardo I SIMARI, and Marcelo A FALAPPA. « Belief revision in structured probabilistic argumentation ». In *International Symposium on Foundations of Information and Knowledge Systems*, pages 324–343. Springer, 2014. 1.2, 1.3

[SSK<sup>+</sup>12] Hitesh SAJNANI, Vaibhav SAINI, Kusum KUMAR, Eugenia GABRIELOVA, Pramit CHOUDARY, and Cristina LOPES. « Classifying yelp reviews into relevant categories », 2012.

[STY17] Mukund SUNDARARAJAN, Ankur TALY, and Qiqi YAN. « Axiomatic attribution for deep networks ». In *International conference on machine learning*, pages 3319–3328. PMLR, 2017. 1.2.2, 1.2.3, 5.1.1

[SVZ13a] Karen SIMONYAN, Andrea VEDALDI, and Andrew ZISSERMAN. « Deep inside convolutional networks: Visualising image classification models and saliency maps ». *arXiv preprint arXiv:1312.6034*, 2013. 1.2.2, 1.2.2, 5.1.1

[SVZ13b] Karen SIMONYAN, Andrea VEDALDI, and Andrew ZISSERMAN. « Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps », 2013. 5.1.1, 6.1

[SZM19] Julia STROUT, Ye ZHANG, and Raymond J MOONEY. « Do human rationales improve machine explanations? ». *arXiv preprint arXiv:1905.13714*, 2019. 2.3, 5.2.2

[SZS<sup>+</sup>13] Christian SZEGEDY, Wojciech ZAREMBA, Ilya SUTSKEVER, Joan BRUNA, Dumitru ERHAN, Ian GOODFELLOW, and Rob FERGUS. « Intriguing properties of neural networks ». *arXiv preprint arXiv:1312.6199*, 2013. 1

[Tab19] Karim TABIA. « Towards explainable multi-label classification ». In *2019 IEEE 31st International Conference on Tools with Artificial Intelligence (ICTAI)*, pages 1088–1095. IEEE, 2019. 4.4.2

[TBH<sup>+</sup>18] Richard TOMSETT, Dave BRAINES, Dan HARBORNE, Alun PREECE, and Supriyo CHAKRABORTY. « Interpretable to whom? A role-based model for analyzing interpretable machine learning systems ». *arXiv preprint arXiv:1806.07552*, 2018. 2.3

[TK07] Grigorios TSOUMAKAS and Ioannis KATAKIS. « Multi-label classification: An overview ». *International Journal of Data Warehousing and Mining (IJDWM)*, 3(3):1–13, 2007. 1
[TKSR16] Jayaraman J THIAGARAJAN, Bhavya KAILKHURA, Prasanna SATTIGERI, and Karthikeyan Natesan RAMAMURTHY. « Treeview: Peeking into deep neural networks via feature-space partitioning ». *arXiv preprint arXiv:1611.07429*, 2016. 1.2.2

[TKV08] Grigorios TSOUMAKAS, Ioannis KATAKIS, and Ioannis VLAHAVAS. « Effective and efficient multilabel classification in domains with large number of labels ». In *Proc. ECML/PKDD 2008 Workshop on Mining Multidimensional Data (MMD'08)*, volume 21, pages 53–59, 2008. 3

[TKV09] Grigorios TSOUMAKAS, Ioannis KATAKIS, and Ioannis VLAHAVAS. « Mining multi-label data ». *Data mining and knowledge discovery handbook*, pages 667–685, 2009. 1.2

[TPJR18] Yuchi TIAN, Kexin PEI, Suman JANA, and Baishakhi RAY. « Deeplest: Automated testing of deep-neural-network-driven autonomous cars ». In *Proceedings of the 40th international conference on software engineering*, pages 303–314, 2018. 1.1

[Tse83] Grigori S TSEITIN. On the complexity of derivation in propositional calculus. In *Automation of reasoning*, pages 466–483. Springer, 1983. 3.2.1, 3.4.1

[TSHW20] Sarah TAN, Matvey SOLOVIEV, Giles HOOKER, and Martin T WELLS. « Tree space prototypes: Another look at making tree ensembles interpretable ». In *Proceedings of the 2020 ACM-IMS on Foundations of Data Science Conference*, pages 23–34, 2020. 1.2.1

[TT04] Gianluca TORTA and Pietro TORASSO. « The role of OBDDs in controlling the complexity of model based diagnosis ». In *Proc. of 15th International Workshop on Principles of Diagnosis (DX-04)*. Citeseer, 2004. 2.2.1

[TV07] Grigorios TSOUMAKAS and Ioannis VLAHAVAS. « Random k-labelsets: An ensemble method for multilabel classification ». In *European conference on machine learning*, pages 406–417. Springer, 2007. 1, 3

[TvdH13] Nikolaj TOLLENAAR and Peter GM van der HEIJDEN. « Which method predicts recidivism best?: a comparison of statistical, machine learning and data mining predictive models ». *Journal of the Royal Statistical Society: Series A (Statistics in Society)*, 176(2):565–584, 2013. 1.1

[TVRFOG18] Ruben TOLOSANA, Ruben VERA-RODRIGUEZ, Julian FIERREZ, and Javier ORTEGA-GARCIA. « Exploring recurrent neural networks for on-line handwritten signature biometrics ». *Ieee Access*, 6:5128–5138, 2018. 1.1

[TXT17] Vincent TJENG, Kai XIAO, and Russ TEDRAKE. « Evaluating robustness of neural networks with mixed integer programming ». *arXiv preprint arXiv:1711.07356*, 2017.

[TYRW14] Yaniv TAIGMAN, Ming YANG, Marc'Aurelio RANZATO, and Lior WOLF. « Deepface: Closing the gap to human-level performance in face verification ». In *Proceedings of the IEEE conference on computer vision and pattern recognition*, pages 1701–1708, 2014. 1.1, 1

[UR16] Berk USTUN and Cynthia RUDIN. « Supersparse linear integer models for optimized medical scoring systems ». *Machine Learning*, 102(3):349–391, 2016. 1.1, 1.3
[VDH20] Sahil VERMA, John DICKERSON, and Keegan HINES. « Counterfactual explanations for machine learning: A review ». *arXiv preprint arXiv:2010.10596*, 2020. 1.2

[vdWRvD<sup>+</sup> 18] Jasper van der WAA, Marcel ROBEER, Jurriaan van DIGGELEN, Matthieu BRINKHUIS, and Mark NEERINCX. « Contrastive explanations with local foil trees ». *arXiv preprint arXiv:1806.07470*, 2018. 1.2

[VdWSNI<sup>+</sup> 14] Stefan Van der WALT, Johannes L SCHÖNBERGER, Juan NUNEZ-IGLESIAS, François BOULOGNE, Joshua D WARNER, Neil YAGER, Emmanuelle GOUILLART, and Tony YU. « scikit-image: image processing in Python ». *PeerJ*, 2:e453, 2014. 1.1

[Vos16] W Gregory VOSS. « European union data privacy law reform: General data protection regulation, privacy shield, and the right to delisting ». *The Business Lawyer*, 72(1):221–234, 2016. 1.1.1

[Vre19] Mihnea Horia VREJOIU. « Neural Networks and Deep Learning in Cyber Security ». *Romanian Cyber Security Journal*, 1(1):69–86, 2019. 1.1

[Wal12] Joel WALMSLEY. Classical Cognitive Science and “Good Old Fashioned AI”. In *Mind and Machine*, pages 30–64. Springer, 2012. 1

[WDGG19] Dennis WEI, Sanjeeb DASH, Tian GAO, and Oktay GUNLUK. « Generalized linear rule models ». In *International Conference on Machine Learning*, pages 6687–6696. PMLR, 2019. 1.1, 1.3

[WGH19] Lior WOLF, Tomer GALANTI, and Tamir HAZAN. « A formal approach to explainability ». In *Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society*, pages 255–261, 2019. 2.2

[WH13] Siert WIERINGA and Keijo HELJANKO. « Asynchronous multi-core incremental SAT solving ». In *International Conference on Tools and Algorithms for the Construction and Analysis of Systems*, pages 139–153. Springer, 2013. 3.3.4

[WHP<sup>+</sup> 18] Mike WU, Michael HUGHES, Sonali PARBHOO, Maurizio ZAZZI, Volker ROTH, and Finale DOSHI-VELEZ. « Beyond sparsity: Tree regularization of deep models for interpretability ». In *Proceedings of the AAAI conference on artificial intelligence*, volume 32, 2018. 1.2.2

[WKQ<sup>+</sup> 08] Xindong WU, Vipin KUMAR, J ROSS QUINLAN, Joydeep GHOSH, Qiang YANG, Hiroshi MOTODA, Geoffrey J MCLACHLAN, Angus NG, Bing LIU, S Yu PHILIP, and OTHERS. « Top 10 algorithms in data mining ». *Knowledge and information systems*, 14(1):1–37, 2008. 1.1

[WMR17] Sandra WACHTER, Brent MITTELSTADT, and Chris RUSSELL. « Counterfactual explanations without opening the black box: Automated decisions and the GDPR ». *Harv. JL & Tech.*, 31:841, 2017. 1.2, 1.2.2, 1.2.2, 3.3.3, 5

[WR15] Fulton WANG and Cynthia RUDIN. « Falling rule lists ». In *Artificial intelligence and statistics*, pages 1013–1022. PMLR, 2015. 5.2.2

[WRDV<sup>+</sup> 17] Tong WANG, Cynthia RUDIN, Finale DOSHI-VELEZ, Yimin LIU, Erica KLAMPFL, and Perry MACNEILLE. « A bayesian framework for learning rule sets for interpretable
classification». *The Journal of Machine Learning Research*, 18(1):2357–2393, 2017. 1.2.1

[WRK<sup>+</sup>17] Stephen F WENG, Jenna REPS, Joe KAI, Jonathan M GARIBALDI, and Nadeem QURESHI. « Can machine-learning improve cardiovascular risk prediction using routine clinical data? ». *PloS one*, 12(4):e0174944, 2017. 1.1

[WXH<sup>+</sup>14] Yunchao WEI, Wei XIA, Junshi HUANG, Bingbing NI, Jian DONG, Yao ZHAO, and Shuicheng YAN. « CNN: Single-label to multi-label ». *arXiv preprint arXiv:1406.5726*, 2014. 4.2

[WYAL19] Danding WANG, Qian YANG, Ashraf ABDUL, and Brian Y LIM. « Designing theory-driven user-centric explainable AI ». In *Proceedings of the 2019 CHI conference on human factors in computing systems*, pages 1–15, 2019. 1

[WZLW21] Zhuo WANG, Wei ZHANG, Ning LIU, and Jianyong WANG. « Scalable Rule-Based Representation Learning for Interpretable Classification ». *Advances in Neural Information Processing Systems*, 34, 2021.

[WZWY19] Ziyu WANG, Xiao ZENG, Jinzhao WU, and Guowu YANG. « A Method for Determining the Affine Equivalence of Boolean Functions ». *IEEE Access*, 7:156326–156337, 2019. 4.3.1

[YCN<sup>+</sup>15] Jason YOSINSKI, Jeff CLUNE, Anh NGUYEN, Thomas FUCHS, and Hod LIPSON. « Understanding neural networks through deep visualization ». *arXiv preprint arXiv:1506.06579*, 2015. 1.2.2

[YRS17] Hongyu YANG, Cynthia RUDIN, and Margo SELTZER. « Scalable Bayesian rule lists ». In *International conference on machine learning*, pages 3921–3930. PMLR, 2017. 2.3, 5.2.2

[Zad96] Lotfi A ZADEH. A note on prototype theory and fuzzy sets. In *Fuzzy Sets, Fuzzy Logic, And Fuzzy Systems: Selected Papers by Lotfi A Zadeh*, pages 587–593. World Scientific, 1996. 1.2.2

[ZAG18] Daniel ZÜGNER, Amir AKBARNEJAD, and Stephan GÜNNEMANN. « Adversarial attacks on neural networks for graph data ». In *Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining*, pages 2847–2856, 2018. 1.1.2

[ZCAW17] Luisa M ZINTGRAF, Taco S COHEN, Tameem ADEL, and Max WELLING. « Visualizing deep neural network decisions: Prediction difference analysis ». *arXiv preprint arXiv:1702.04595*, 2017. 1.2.2

[ZF14] Matthew D ZEILER and Rob FERGUS. « Visualizing and understanding convolutional networks ». In *European conference on computer vision*, pages 818–833. Springer, 2014. 1.2, 1.2.2, 1.3, 5.1.1

[ZH21] Qingyuan ZHAO and Trevor HASTIE. « Causal interpretations of black-box models ». *Journal of Business & Economic Statistics*, 39(1):272–281, 2021. 2.1
[ZKL<sup>+</sup>16] Bolei ZHOU, Aditya KHOSLA, Agata LAPEDRIZA, Aude OLIVA, and Antonio TORRALBA. « Learning deep features for discriminative localization ». In *Proceedings of the IEEE conference on computer vision and pattern recognition*, pages 2921–2929, 2016. 1.2, 1.2.2, 1.2.3, 1.3

[ZVR<sup>+</sup>17] Muhammad Bilal ZAFAR, Isabel VALERA, Manuel RODRIGUEZ, Krishna GUMMADI, and Adrian WELLER. « From parity to preference-based notions of fairness in classification ». *Advances in Neural Information Processing Systems*, 30, 2017. 1.1.2, 1.2, 1.3

[ZVRG17] Muhammad Bilal ZAFAR, Isabel VALERA, Manuel Gomez ROGRIGUEZ, and Krishna P GUMMADI. « Fairness constraints: Mechanisms for fair classification ». In *Artificial intelligence and statistics*, pages 962–970. PMLR, 2017. 1.1.2

[ZYMW19] Quanshi ZHANG, Yu YANG, Haotian MA, and Ying Nian WU. « Interpreting cnns via decision trees ». In *Proceedings of the IEEE/CVF conference on computer vision and pattern recognition*, pages 6261–6270, 2019. 1.2.2

[ZZ05] Min-Ling ZHANG and Zhi-Hua ZHOU. « A k-nearest neighbor based algorithm for multi-label classification ». In *2005 IEEE international conference on granular computing*, volume 2, pages 718–721. IEEE, 2005. 2
Résumé

Cette thèse étudie une méthode d'explicabilité qui allie à la fois le caractère "agnostique" des méthodes numériques et qui propose des explications plus "rigoureuses" qui caractérisent les explications symboliques. Le but étant d'expliquer les prédictions des techniques de classification mono-étiquette et multi-étiquettes. Plusieurs contributions sont apportées dans cette thèse. Premièrement, nous avons travaillé sur le cas mono-étiquette. Nous avons proposé une approche qui va de l'encodage en représentation symbolique du modèle dont on souhaite expliquer les prédictions à la génération d'explication basée sur un oracle SAT. L'idée est de prendre un classifieur, avec une instance, et de produire une formule propositionnelle que nous utiliserons pour générer nos explications. L'inconsistance de cette formule permet d'expliquer les prédictions. Nous considérons les deux cas où nous pouvons avoir la représentation logique du modèle dans son ensemble ou une approximation basée sur un modèle de substitution. Nous nous intéressons à deux types complémentaires d'explications symboliques : les *raisons suffisantes* qui correspondent à un sous-ensemble minimal de l'entrée conduisant à une prédiction spécifique et les *contrefactuelles* qui correspondent à un sous-ensemble de l'entrée permettant de déterminer les modifications minimales à apporter pour obtenir une prédiction différente. Deuxièmement, nous avons proposé des propriétés à considérer afin de prioriser et sélectionner les explications en évaluant leur pertinence ainsi que celle des variables les composants. Par la suite, nous nous sommes intéressés à l'explication des prédictions multi-étiquettes. Nous avons proposé des explications multi-étiquettes à différents niveaux de granularité et étudié la combinaison d'explications mono-étiquette ainsi que les relations structurelles entre classes comme moyen de les générer. Enfin, nous nous sommes intéressés aux scores d'importance au niveau des caractéristiques pour déterminer dans quelle mesure chacune contribue à la sortie d'un modèle multi-étiquettes. Cette contribution examine deux possibilités différentes d'utiliser des méthodes existantes pour le cas mono-étiquette comme oracles ou d'utiliser des attributions de caractéristiques obtenues à partir d'explications symboliques. Afin d'évaluer la qualité des attributions de caractéristiques, nous étendons les propriétés de sensibilité, de stabilité des données au cas multi-étiquettes en plus d'une nouvelle propriété spécifique à la classification multi-étiquettes que nous appelons corrélation label-explication.

**Mots-clés:** IA explicable, Explications symboliques, Explications basées-score, Modèle-agnostique, Classification multi-étiquettes, Satisfiabilité booléenne, Attribution de caractéristique.
Abstract

This thesis studies the problem of explaining individual predictions of black-box machine learning
models. This problem is addressed in both single and multi-label classification. Firstly, we introduce an
explanation approach representing a combination of SAT solving and numerical measures to develop a
model-agnostic method for providing both symbolic and score-based explanations. The idea is to take a
single-label classifier, together with an instance, and produce a propositional formula that we will use to
generate our explanations. We consider both cases where we can have the logical representation of the
model as a whole or an approximation based on a surrogate model. In the second case, a crucial compo-
nent of the proposed approach is to approximate the model with another (simpler) one that does admit a
tractable logical representation to efficiently enumerate explanations. To comply with the original pre-
dictor, the selected surrogate model needs to ensure fidelity. Subsequently, this trained model is used to
generate symbolic and numerical explanations. In a second time, we consider a SAT framework with the
aim of using SAT solvers as the problem solving engine. Given an unsatisfiable formula corresponding
to a negative prediction, modern SAT solvers are able to report the cores generating an inconsistency.
In this contribution, we provide two complementary types of symbolic explanations of unsatisfiability
called sufficient reasons and counterfactuals centered around Minimal Unsatisfiable Subsets (MUS)
and Minimal Correction Subsets (MCS) respectively. Secondly, we have worked on defining measures
of the quality of an explanation and of a variable contribution to properly assess how relevant they are
as it becomes necessary to focus on those providing more insights. Next, we have worked on defining
possible explanation mechanisms to explain the outcomes of multi-label classifiers. We have introduced
explanations at different granularity levels which go from structural relationships between labels to the
selection of features. Finally, we were interested in feature-level importance scores for how much a given
input feature contributes to a multi-label model's output. This contribution looks into two different pos-
sibilities of using existing methods for single-label as oracles or using feature attributions obtained from
symbolic explanations. In order to evaluate the quality of feature attributions, we extend the properties
of sensitivity, data-stability to the multi-label setting in addition to a new property specific to multi-label
classification we called label-explanation correlation.

**Keywords:** eXplainable AI (XAI), Symbolic explanations, Score-based explanation, Model-agnostic, Multi-label classification, Satisfiability testing, Feature attribution.