Test effects of making kernel watchdog more patient than NFS timeout on api-XX nodes

#### Goal:

Determine if making the watchdog more patient can change the failure mode from a kernel panic and reboot to a failed synchronous write operation.

Determine if a failed NFS write is more or less impactful to our Rails app on `api-XX` VMs than a kernel panic and automatic reboot.  (Example: If the filesystem becomes read-only, then the panic+reboot is preferable because recovery requires no manual intervention.)

#### Method outline:

* Check that the NFS timeout really does follow the arithmetic outlined below (which is based on the nfs manpage, not based on empirical testing).  The two relevant mount options are: `timeo=50` (50 centiseconds = 5 seconds) and `retrans=2`.  The manpage says each retry linearly adds another `timeo` to the timeout, and 2 retries means 3 tries in total.  So: 1st try waits 5 seconds, 2nd try waits 10 seconds, 3rd try waits 15 seconds, then fail after a total patience of 5+10+15=30 seconds.
* On a non-production VM, try to do a *synchronous write* to an NFS file while sabotaging that write in some way that the NFS client is unaware of.  (Maybe a client-side iptables rule to drop outgoing or incoming NFS traffic?  Disabling the NIC would let the kernel know to sever the TCP connection, so that might not fail in the intended way.)  This test aims to synthetically confirm that we can reproduce the observed behavior.
* Adjust the kernel watchdog timeout (2 * `kernel.watchdog_thresh` = 20 seconds) to be more patient than the NFS timeout (30 seconds I think, see above).
* Repeat the synthetic test.  Does the VM fail the write instead of panic and reboot?  Is the NFS volume still mounted in read-write mode?
* Revert the sabotage.  Does the VM's NFS mount become usable again?
* If the above tests were done on one of our `api-XX` VMs, is the Rails app still functional without rebooting the VM?

#### Background:

In #7872 we have been tracking GCP VM spontaneous reboots.  A subset of those reboots are `soft lockup` kernel panics.  In the weeks since we started tracking these unplanned reboots, this particular flavor has only occurred on `api-XX` and `web-XX` VMs.  These VMs regularly use NFS volumes, and that might be relevant.

We have ample evidence that the GCP platform sometimes drops network connectivity to some or all network peers, often lasting for tens of seconds before recovering.  Some of these `soft lockup` events show signs of such a network connectivity disruption preceding the kernel panic.  So far **all** of these `soft lockup` events share [nearly identical kernel stack traces](https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/7872#note_227714232), suggesting these events have a common cause.  At the time of the panics, the hung process that was hogging a CPU (implicitly preventing that CPU's watchdog thread from running) was stalled waiting on a synchronous `write` system call.

As a [working hypothesis](https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/7872#note_227712046), that hung process's `write` syscall may leave the process in uninterruptible sleep, preventing other processes (including watchdog) from being scheduled on that vCPU.  We do not know what that `write` syscall's file descriptor refers to.  It could point to a file on NFS, a file on ext4, or a network socket.  Because this pathology has so far only affected hosts using NFS, that possibility may be more likely.

If the hung process is stalled while writing to NFS, then if we adjust the watchdog timer to be more patient than the filesystem's timeout, then the outcome should change from a kernel panic to a failed write.

A write failure should be logged by the kernel (i.e. reliably detectable and more clear to interpret than a kernel panic).  But we need to know if the failed write will lead to longer lasting application problems, since it would not automatically reboot the VM.

This issue is to test the effects of a failed NFS write on one of our `api-XX` VM, and determine if this outcome is:
* diagnostically useful
* preferable to a reboot from an availability standpoint


##### What is a `soft lockup`?

A `soft lockup` kernel panic is triggered by the kernel detecting that at least one of its CPUs appears to be hung.  The detection mechanism is that each processor (CPU core or vCPU) has a dedicated watchdog thread that resets a count-down timer each time it runs.  If any processor fails to run that watchdog thread before the timer reaches zero, the kernel takes this as a sign of extreme duress and possibly a bug, since normally the scheduler would easily be able to issue a time slice to the watchdog thread even under heavy load.  So the kernel panics, calling it a `soft lockup` event.

Story points: 2.0