Make sure we apply attributes to correct places.

Some attributes make sense only on the function or on the call site, but not
both. Make this distinction here.