Finally done this correctly.
Only my app specific files are tracked now