logsumexp([-inf, ..., -inf]) is nan. Check for that and make it -inf.