Skip all gradients except linear for now