Motion Mask is a feature to eliminate an area of the video from Motion Detection. It does not affect what you can see in the Live Video.
You may have a ceiling fan or tree branches that are constantly moving and causes unwanted motion detection recordings. By masking this area, the Motion Detection Software will ignore this motion.
You can access this in Advanced Settings, Sensitivity Setting, Motion Mask.