Bug1001455 Remove some gradients to save memory

Remove some gradients to save memory