If the amp power is too low, then the answer is yes. The sound will be noticeably quieter and possibly have a lot of white noise. Using good-quality headphones with low output impedance and the source is good, so using a headphone splitter should not reduce audio quality. You might hear a slight difference, but it would be hard to notice.