This is true, for example, of Great Britain, where trade with colonies and commonwealth partners has declined substantially relative to trade with former rival imperial powers in Europe. It is also dramatically true for the transition countries of Central and Eastern Europe and the former Soviet Union. Since 1990, trade among them has declined enormously, while trade with the rest of the world has picked up substantially. Obviously, imperialism is a matter of public policy. So the short answer to the overarching question of why the 20th century was unkind to imperialism is that public policy changed.
Tastes are probably part of the answer.
Just as moral revulsion against slavery was critical to its suppression in the 19th century, revulsion at the great carnage of war and the brutality of oppression have helped turn the tide against imperialism in the 20th century. Mass communications that graphically portray carnage and brutality have contributed to the change in public attitudes. Perhaps more important, however, is the shift in technology that has made imperialism an inefficient, if not counterproductive, means of improving economic welfare. The 20th century has been a very uncomfortable time for imperialists to seek to impose their will on other peoples either for economic gain or for other reasons. Unwelcome efforts to exert control over an alien people, especially in the face of armed opposition, tends to be very expensive in blood and treasure.