Explainable AI in the Banks and Finance industry is not just about giving justification for a model's decision. It explains each element of the solution process. Such as Data, Algorithm, Model, and Output.