I'd say that the vast majority of us are not into children. As with any group of people, there are various interests and activities that might be found. Most people's plushophilia has little to do with children, and most who are into children would probably be bored with a plushie. There just isn't much in common as far as most plushophiles are concerned. Children are yucky, smooth and make noise.
Plushies are cute, furry and quiet.
Many plushophiles simply prefer the texture of fur over the texture of smooth skin. Plushies are often not designed to be children's toys. Some are made to be collected. Others are just made to look good, or to bring happiness to whoever needs it. Most plushies are bought by adults and for adults.