Engine: fixed two exceptional cases in alpha blender

This fixes two errors:
1. A division by zero occuring if the custom alpha supplied is of very low value;
2. A final alpha overflow that caused weird colors appear with particular combination of input parameters.