Code cleanup: use bool instead of int in mask module

git-svn-id: https://svn.blender.org/svnroot/bf-blender/trunk@61000 954f8c5b-7b00-dc11-b283-0030488c597c