“We dug coal together,” has been Raylan's way of explaining his connection to all things Harlan over the course of the series. The line has always been an admission and sidestep at the same time. It's his answer to Ava way back in the pilot, when he shoots Boyd but won't leave him to die. Yes, Raylan's lying to Boyd.