The difference is mostly not in the compile time per cpp but in the number of cpps that recompile if I change some header file. If VoxelCluster includes VoxelRenderData and I do a change in voxelrenderdata.h about half the cpps need to recompile because voxelcluster is included everywhere. None of these cpps have done anything with voxelrenderdata.

That is not a reason to change EVERY member to a pointer. But you should take care of includes in headers that would propagate to a huge amount of classes that don't need the include.