Only increment the scandir baton when recursing

Fix #285 but needs testing for more complex directories