Only modify egg directories that are part of the package being built

CC @talumbau @ilanschnell 

Opening a pull request because I didn't test it yet.