Fix trivial cast detection: builders are problematic