This is basically a test if points C and D are on different sides of a line trough A and B. A->B represents the diagonal of a quad and for it to be convex the other two nodes need to be on different sides of that diagonal. 
In 3D space you can think of the line extended to a plane in z-direction. Since we don't know how the quad (and therefore the other two points) are located in 3D space, we rotate this plane (you can think of this as testing with a plane in xy, yz and xz). If we find any combination that such a plane divides C and D then the quad is convex.
This is a test needed for many geometrical geometrical problems, so I put it here in case it is needed elsewhere.