I learned that I have to remove the files first from the cache, then git will ignore them