The actual difference is in the materials they handle. A plumber works mostly on commercial and residential drain systems that include copper and PVC piping. Pipefitters work with specialized equipment, high-pressure systems, and hazardous materials.
Steel, stainless, glass, and other types of pipes will fall under the pipefitter's repertoire, where a plumber won't have to deal with any of these types of piping or fittings.