We can only run diff on directories