I agree - this is more a practical solution to make sure that we refresh the most likely things w/o slowing down Spark. As per your bug, there are more sophisticated ways of doing this.

In other environments there are APIs to subscribe to push notifications for file changes. Polling for changes - periodically or lazily - are more resource intensive and less timely then file watching APIs.