Just use git rm and ignore the commit conflict for a file that is not needed on another branch