Fix #1276: Do not check sizes for Scala versions not in PR matrix.

Otherwise, they become out-of-sync, and hence completely useless, pretty easily.