Making the min improvement really small and picking random starting points for the gradient descent seemed to help a lot.