Fix over-aggressive removal

This change adds back a bit of the previous commit that was removed to
aggressively.
Merge 60962838-specify-buildpack to master

[Completes #60962838]