Validate SmartHTTP::PostUploadPack

~Conversation: https://gitlab.com/gitlab-org/gitaly/issues/219

See the [Migration Process documentation](https://gitlab.com/gitlab-org/gitaly/blob/master/doc/MIGRATION_PROCESS.md#acceptance-testing-acceptance-testing) 
for more information on the Acceptance Testing stage of the process.

Feature Toggle Environment Variable: `GITALY_POST_UPLOAD_PACK`

--------------------------------------------------------------------------------

- [x] [Chef recipes](https://dev.gitlab.org/cookbooks/chef-repo) to enable/disable this feature https://dev.gitlab.org/cookbooks/chef-repo/merge_requests/684
- [x] [Grafana dashboard](https://gitlab.com/gitlab-org/grafana-dashboards/) for monitoring https://gitlab.com/gitlab-org/gitaly-dashboards/merge_requests/5
- [ ] Environments
    - [x] `dev.gitlab.org` https://dev.gitlab.org/cookbooks/chef-repo/merge_requests/685
    - [x] Staging https://dev.gitlab.org/cookbooks/chef-repo/merge_requests/684
    - [x] `gitlab.com` https://dev.gitlab.org/cookbooks/chef-repo/merge_requests/696 https://docs.google.com/document/d/1SmZvlT7D1dXsEtRPZc0xWR0jGmPmLztlGkG4_L9DQeU/edit
- [ ] Test Results (post as comments on this issue)
    - [ ] Did the migration perform as expected? 
    - [ ] Did the code have reasonable performance characteristics?
    - [ ] Did error rates jump to an unacceptable level?
- [ ] Have the changes been rolled back pending final review?
- [ ] Runbook Added (link to MR)
- [ ] Prometheus Alerts Added (link to MR)

-----------------------------------------------------------------------------------

We now (9.1) have a feature flag GITALY_POST_UPLOAD_PACK in gitlab-ce. We should validate that it works:

- [x] test in omnibus nightly
- [x] test in RC omnibus package
- [x] test in staging (apply setting, do manual clones, look at prometheus counters for errors and to see if the RPC is being used)
- [x] test in production for a few hours (same as staging)
- [ ] test in production for a week
- [ ] enable feature flag by default in omnibus

----
## Staging steps

- update `gitlab-staging-worker` role (make sure change is on chef.gitlab.com, not just in the chef-repo)
- `knife ssh -C 1 role:gitlab-staging-worker 'sudo chef-client; sleep 60; sudo gitlab-ctl term unicorn; echo done $?'` (`-C 1` is probably too slow for production)
- `knife ssh role:gitlab-staging-worker 'for p in $(pgrep -f "unicorn master"); do ps -o pid,etime,args -p $p; sudo cat  /proc/$p/environ | xargs --null --max-args=1; done' | grep GITALY`
 - check [gitaly staging prometheus dashbord](https://performance.gitlab.net/dashboard/db/gitaly?orgId=1&var-job=gitaly-staging)
- check [Sentry for staging](https://sentry.gitlap.com/gitlab/staginggitlabcom/)
- manual tests
- revert config change

## Local omnibus steps

In gitlab.rb:

```ruby
gitlab_rails['env'] = {'GITALY_POST_UPLOAD_PACK' => '1'}
```

To apply or revert:

```
sudo gitlab-ctl reconfigure
sleep 60 # wait for unicorn to reload
sudo gitlab-ctl term unicorn
# test environment of unicorn master. should print a match
for p in $(pgrep -f 'unicorn master'); do ps -o pid,etime,args -p $p; sudo cat  /proc/$p/environ | xargs --null --max-args=1 | grep GITALY; done
```

The rigmarole with `gitlab-ctl term` is because of https://gitlab.com/gitlab-org/omnibus-gitlab/issues/2002

Story points: 3.0