Use appropriate buffer for QgsGeometry::XXXContainedInLine. Fix #10999.

We have reduced the problem by reducing the size of the buffer used in QgsGeometry::XXXContainedInLine functions.
The error may still occur depending on:
   the accuracy of the projection used,
   the distance between the object to modify and the nearest point.
So we caught the error, stop capturing and displaying a more specific message.
But as far as I understand, geomDigits() returns number of digits before the dot, so the results would be:
~0.00001 for mercator
~0.1 for lat/lon