Obviously, the material that is falling on to the compact object has to come from somewhere. Therefore, the relativistic stars must be stealing the material from someone else. Most X-ray sources correspond to binary systems in which the compact object is dragging material from some other star. Systems composed of two stars are called binaries.
Most stars have binary companions.
Isolated stars like the Sun are the exception rather than the norm. But only a small proportion of binary systems will evolve into X-ray binaries. The in-falling matter forms an accretion disc around the neutron star before being accreted. The neutron star at the center of the accretion disc is too small to be seen. However, their X-ray emission is comparatively very small.