Update BGD AllReduce to make the program can end normally when failure happens at the final iteration