doc: update/clarify behaviour of mask in floodFill

Clarify how the mask parameter is set on output and how this is
affected by the flags parameter.

resolves Feature #2942
Merge pull request #2738 from kevmitch:2.4