The CPU code do that too. We don't want the exp to explode and cause instability do to numeric precision. That is why we reduce the value of the inside of the exp. This op is there for numeric stability and this is how it do it.