Allow setting gamma at the layer level (#4763)