Genre: Forum

I can't confirm that tuning to better CV reliably results in decreased leaderboard score, but I have been suprised at the lack of leaderboard improvement when improving CV scores. There have been a few times in this contest where I have followed up a hunch with a decent amount of work and repeated CV (with promising results), only to be hit with some terrible leaderboard scores. Given that this contest is using a relatively small fraction of the test data to calculate the leaderboard, I'm trying to slowly improve my CV and do my best to ignore the leaderboard. Much easier said than done! Interesting! I went at it from the opposite direction, first building and tuning a model for psychopathy and then applying the same code, without further optimization but with target variable and training file changed appropriately, to each of the personality traits in turn. CV scores obtained in this way were variable: with an average of 0.8535. The corresponding leaderboard score was 0.86048. Just for fun I tried a second, linear model that predicted each of the 7 traits by using only the (first model) results for the other 6 as the predictors. This gave a leaderboard score of 0.84888, only slightly worse than the random forest benchmark. For building a validation set you can choose just few instances. It isn't a big surprise that there is much variance between all sets. So i just tried to make an all round model... I'm really curious how the final scores will be. In contests like that the biggest shuffling in top scores i saw was a 10h taking first... With the 6th, 11th, 24th and 30th on the public board finishing 1 through 4 on the private, I'd say you called it just about right, Leustagos. Looks like about 6 to 12 entries were enough. Congratulations to y_tag on the win!