doc: update/clarify behaviour of mask in floodFill

Clarify how the mask parameter is set on output and how this is
affected by the flags parameter.

resolves Feature #2942