BUG/CLN: Cleanup sanitize column and abstract broadcasting

Also fixes small bug from #5321