Bug 878062 - Don't call TryOptimizeToImageLayer if we are definitely not going to optimize to an image layer, since it uses lots of memory. r=mattwoodrow