Well, I think you're right that it looks like you shouldn't build there. It just looked OK on snow, so I didn't realize that on temperat and desert it might look bad. 
In the end, it's ambiguous. I guess what's important is that 
a) it doesn't look ugly or obviously wrong
b) it is not counter-intuitive to the player 
If the player would be surprised he could build there, then maybe we should block it.