Leave size as -1 for updated folders during a shallow scan

The tests seem to be temperamental, a bad effect of using time() :(

I'm not sure if this resolves the problem, but the code looks right to me.

@icewind1991 @jvillafanez