Checking the numerical gradient reveals a small error in my gradient calculation...