A100 runners down: apt-get install nvidia-docker2, Could not get lock /var/lib/dpkg/lock-frontend
Labels: triaged, module: infra

> NOTE: Remember to label this issue with "`ci: sev`"

## Current Status
mitigated

## Error looks like
```
2023-09-08T08:40:51.0035073Z == Installing nvidia container toolkit for ubuntu20.04 ==
2023-09-08T08:40:51.0039409Z + sudo apt-get install -y nvidia-docker2
2023-09-08T08:40:51.0725043Z E: Could not get lock /var/lib/dpkg/lock-frontend. It is held by process 3074496 (apt-get)
2023-09-08T08:40:51.0725789Z E: Unable to acquire the dpkg frontend lock (/var/lib/dpkg/lock-frontend), is another process using it?
2023-09-08T08:41:01.5406321Z ##[warning]Attempt 1 failed. Reason: Child_process exited with error code 100
```

also

```
2023-09-08T09:10:53.7042430Z == Installing nvidia container toolkit for ubuntu20.04 ==
2023-09-08T09:10:53.7045422Z + sudo apt-get install -y nvidia-docker2
2023-09-08T09:10:53.7771078Z Reading package lists...
2023-09-08T09:10:54.0655363Z Building dependency tree...
2023-09-08T09:10:54.0661679Z Reading state information...
2023-09-08T09:10:54.3444752Z nvidia-docker2 is already the newest version (2.13.0-1).
2023-09-08T09:10:54.3445264Z The following packages were automatically installed and are no longer required:
2023-09-08T09:10:54.3445840Z   libatasmart4 libblockdev-fs2 libblockdev-loop2 libblockdev-part-err2
2023-09-08T09:10:54.3446423Z   libblockdev-part2 libblockdev-swap2 libblockdev-utils2 libblockdev2
2023-09-08T09:10:54.3446989Z   libexpat1-dev libmm-glib0 libnspr4 libnss3 libnuma1 libparted-fs-resize0
2023-09-08T09:10:54.3458106Z   libpython3-dev libpython3.8-dev libudisks2-0 usb-modeswitch
2023-09-08T09:10:54.3458500Z   usb-modeswitch-data
2023-09-08T09:10:54.3458880Z Use 'sudo apt autoremove' to remove them.
2023-09-08T09:10:54.6324446Z 0 upgraded, 0 newly installed, 0 to remove and 72 not upgraded.
2023-09-08T09:10:54.6324867Z 1 not fully installed or removed.
2023-09-08T09:10:54.6325313Z After this operation, 0 B of additional disk space will be used.
2023-09-08T09:10:54.6442593Z Setting up grub-efi-amd64-signed (1.187.3~20.04.1+2.06-2ubuntu14.1) ...
2023-09-08T09:10:54.7556459Z debconf: unable to initialize frontend: Dialog
2023-09-08T09:10:54.7557035Z debconf: (Dialog frontend will not work on a dumb terminal, an emacs shell buffer, or without a controlling terminal.)
2023-09-08T09:10:54.7557494Z debconf: falling back to frontend: Readline
2023-09-08T09:10:55.0194934Z Configuring grub-efi-amd64-signed
2023-09-08T09:10:55.0195638Z ---------------------------------
2023-09-08T09:10:55.0195818Z 
2023-09-08T09:10:55.0196068Z The GRUB boot loader was previously installed to a disk that is no longer 
2023-09-08T09:10:55.0196580Z present, or whose unique identifier has changed for some reason. It is important
2023-09-08T09:10:55.0197051Z to make sure that the installed GRUB core image stays in sync with GRUB modules 
2023-09-08T09:10:55.0197535Z and grub.cfg. Please check again to make sure that GRUB is written to the 
2023-09-08T09:10:55.0197955Z appropriate boot devices.
2023-09-08T09:10:55.0198129Z 
2023-09-08T09:10:55.0198284Z   1. /dev/sda (1099511 MB; ) on 1099511 MB PersistentDisk
2023-09-08T09:10:55.0198674Z   2. /dev/sda15 (111 MB; /boot/efi) on 1099511 MB PersistentDisk
2023-09-08T09:10:55.0198981Z   3. none of the above
2023-09-08T09:10:55.0199144Z 
2023-09-08T09:10:55.0199318Z (Enter the items you want to select, separated by spaces.)
2023-09-08T09:10:55.0199534Z 
2023-09-08T09:20:54.1060071Z /home/weiwangmeta/actions-runner/_work/_actions/nick-fields/retry/3e91a01664abd3c5cd539100d10d33b9c5b68482/dist/index.js:3193
2023-09-08T09:20:54.1060859Z             throw err;
2023-09-08T09:20:54.1061927Z             ^
2023-09-08T09:20:54.1062665Z 
2023-09-08T09:20:54.1062890Z Error: kill EPERM
2023-09-08T09:20:54.1063398Z     at process.kill (node:internal/process/per_thread:221:13)
2023-09-08T09:20:54.1064714Z     at killPid (/home/weiwangmeta/actions-runner/_work/_actions/nick-fields/retry/3e91a01664abd3c5cd539100d10d33b9c5b68482/dist/index.js:3203:17)
2023-09-08T09:20:54.1066048Z     at /home/weiwangmeta/actions-runner/_work/_actions/nick-fields/retry/3e91a01664abd3c5cd539100d10d33b9c5b68482/dist/index.js:3180:21
2023-09-08T09:20:54.1066721Z     at Array.forEach (<anonymous>)
2023-09-08T09:20:54.1067816Z     at /home/weiwangmeta/actions-runner/_work/_actions/nick-fields/retry/3e91a01664abd3c5cd539100d10d33b9c5b68482/dist/index.js:3178:23
2023-09-08T09:20:54.1068574Z     at Array.forEach (<anonymous>)
2023-09-08T09:20:54.1069912Z     at killAll (/home/weiwangmeta/actions-runner/_work/_actions/nick-fields/retry/3e91a01664abd3c5cd539100d10d33b9c5b68482/dist/index.js:3177:27)
2023-09-08T09:20:54.1071251Z     at /home/weiwangmeta/actions-runner/_work/_actions/nick-fields/retry/3e91a01664abd3c5cd539100d10d33b9c5b68482/dist/index.js:3168:13
2023-09-08T09:20:54.1073373Z     at ChildProcess.onClose (/home/weiwangmeta/actions-runner/_work/_actions/nick-fields/retry/3e91a01664abd3c5cd539100d10d33b9c5b68482/dist/index.js:3224:17)
2023-09-08T09:20:54.1074270Z     at ChildProcess.emit (node:events:513:28) {
2023-09-08T09:20:54.1074850Z   errno: -1,
2023-09-08T09:20:54.1075331Z   code: 'EPERM',
2023-09-08T09:20:54.1075823Z   syscall: 'kill'
2023-09-08T09:20:54.1076208Z }
```

## Incident timeline (all times pacific)
Sep 7 - 4:35PM EDT Began @FindHao job at https://github.com/pytorch/pytorch/actions/runs/6114533752 is first to fail this way
Sep 8 - 8:56AM EDT Detected when PT2 oncall checked benchmark results
Sept 8 - 11AM Created #108871 AMI pinning PR and started testing
Sep 8 - 1PM Tried to restart A100 workers (without success) 
Sept 10 7AM EDT Deployed pinning of AMI : https://github.com/pytorch/pytorch/pull/108871 
Sep 10 - 2PM EDT Deployed fix https://github.com/pytorch/test-infra/pull/4562
Sep 10 - 6PM  EDT Deployed followup https://github.com/pytorch/test-infra/pull/4563

## User impact
No A100 benchmark runs can run in CI

## Root cause
Unknown

## Mitigation
Unknown

## Prevention/followups
Unknown