Q:

The level set of a smooth function

Let $f$ be a smooth function on a manifold $M$. Fix a point $p\in M$ and let $df\in T^\ast_pM$ be the differential of $f$ at $p$. I read that the subspace of $T_pM$ of vectors $X$ such that $df(X)=0$ consists of all vectors tangent to curves lying in the surface $\{f=\text{constant}\}$ at $p$. I am hoping that someone can explain to me why this is true.  I do not see it geometrically.  Having $df(X)=0$ means that the directional derivative of $f$ in the direction of $X$ is zero.  What can I conclude from here?  The book then reads that "$df$ can thus be thought of as a normal to the surface $\{f=\text{constant}\}$ at $p$." I would really appreciate if someone could explain to me how this holds.  

A:

As you wrote, for the purpose of answering your question, you should think of the quantity $df(X)$ (at a point $p$) as the directional derivative of $f$ in the direction $X$. This is zero if $f$ is constant in the direction $X$, or in other words if $X$ is tangent to the level set of $f$ through $p$. One way to make this more convincing is to think about a parametrized curve $$p(t):(-1,1) \rightarrow M \quad \hbox{with $p(0)=p$, $p'(0)=X$, and image contained in the level set.}$$ It should be intuitive that the vector $X=p'(0)$ is tangent to the level set through $p$, and on the other hand by the chain rule $$df(X)=df(p'(0))=(f \circ p)'(0)=0$$ since $f \circ p$ is constant. 
The gradient of $f$ is the vector field $\mathrm{grad}(f)$ uniquely determined by  $$\langle \mathrm{grad}(f),X \rangle=df(X)$$ for all vector fields $X$, where we are assuming we have some fixed Riemannian structure $\langle \cdot, \cdot \rangle$ allowing us to do geometry on $M$. You can therefore think of the gradient field as being normal to the level sets of $f$ since our condition translates precisely to it being perpendicular to tangent fields $X$ to level sets.