Remove affinity hacks

See https://github.com/chocolate-doom/chocolate-doom/issues/484.