Apply 'extra' refs to outer refinement

This is an idea I had for dealing with #167. The idea is to simply apply 'extra' predicates to the outer refinement. This solution still drops any RPoly refs, which could still be an issue. We could either do something smarter to handle this case, or throw an error.