a fixed holdout set that we fix in time for the history of the universe does make that test error go down but that's also not generalization error alright if you do a holdout split or you take a train set and you take a holdout set and then you just fix that holdout set forever maybe what's happening here is you have this these giant models have enough fluctuations in them that they can you now over you could actually leak a lot to the test set and over fit to this one holdout set that you fixed forever so this leads to a question this