Match the lighting/alpha logic in the sw transform

This just makes the logic match the sw transform logic as far as I could tell.

Should be similar to #597, but handle a few cases more like sw does currently.

-[Unknown]