Okay, I'm not sure what's the best way to handle that in general, so that's good to have a discussion about that.

I usually provide tarballs with what's needed to run the software, not to develop with it / runs the tests. It seems that having a makefile to check that everything is installed well and works as expected can be useful (for instance for some source distributions) so I'm +1 to include the makefile here.