Disable caching of visibility.

The testcase in pr14929 shows that this is extremely hard to do. If we choose
to apply the attribute, that causes the visibility of some decls to change and
that can happen really late (during codegen).

Current gcc warns and ignores the attribute in this testcase with a warning.
This suggest that the correct solution is to find a point in the compilation
where we can compute the visibility and
* assert it was never computed before
* reject any attempts to compute it again in the future (with warnings).