Fix wrong clip being used for alpha in some RGB layer routines

Reported [here](http://forum.doom9.org/showpost.php?p=1673438&postcount=705).