Don't infer Base.return_type unless the input types are known

This is stricter than necessary but should work.