calculate Hessian on unconstrained space