Remove address specialization for interpreter

This turns out not to be of much gain, since benchmarks show the same
numbers before and after this change (some tiny improvements in certain
places, tiny degrations in others). Because of the added complexity it
does have, we might as well remove it for now and review other
techniques if we want to speed up the interpreter significantly.