If you believe the former, insider trading should not bother you.
Stocks are just bits of paper whose rise and fall have little to do with their underlying worth.
The game is rigged in favour of those with inside information just like the odds favour the casino over the roulette player. If you believe the latter, insider trading should bother you. It means that a select few will always get rich at the expense of the ordinary investor. It means that no amount of diligence and research will pay off because someone with advance knowledge about profits or an acquisition or an executive change will always come out on top. If you believe the market is always rigged against you, you won't bother playing.