Setting correct ownership of the gradient data