docs: update Makefile.am

Use $(top_srcdir) to reference the sources and update the test environment so
that gtkdoc-check needs less guessing.