People want to influence elections and public policy because the power of government has grown so enormously. Government officials have the power to grant favors and to crush opponents almost at will. Of course people are willing to spend fortunes to gain influence. If you really want to decrease the flow of money into politics, strip government of much of its power. The flood of cash will dry up all by itself.
Money is apparently a very bad thing.
Money is a uniquely corrupting influence that draws politicos from the straight and narrow like schoolkids to a crack dealer. The next thing you know, they're selling their votes for another fix. Arguing that money isn't part of speech is as sensible as arguing that paper, or ink, or broadcast facilities aren't speech. Something that overt would likely scare off even the most credulous good-government supporters and would certainly run afoul of First Amendment protections. In general, banning soft money would tip the scales toward incumbents.