The main problem with idealism is that it is intrinsically incompatible with common binary logic, producing paradoxes and contradictions that serve fine to give a touch of profundity to a work of art, but cause much inconvenience for an idealistic-minded scientist. However, postulating the existence of that universal mind differs little from postulating the diversity of material things, and the very distinction between idealism and materialism gets lost. On the other side, this may result in more mental lability, and many problems could be much easier formulated by idealists, despite all their inability to resolve them. The impossibility of consistent idealism results in that 'pure' idealists could only be known in psychiatry rather than philosophy. One has always to deal with an eclectic mixture of materialist and idealist views, and a particular person can often adopt an idealist stand in one or two problem, remaining materialistic in most others.
All idealists use to eat and drink, breathe air and live somewhere.
Surely, they can call all that mere imagination, provided they are not robbed of it. Thus, there can be no grounds for the existence of the world as Matter, and no logic can be decisive in determining the validity of any statement. One has always appeal to 'out there', being ready for more surprises to come with new experience. One cannot 'derive' the world from a single idea, as many idealists are apt to. Merely postulating Matter is not enough to express the integrity of the World.