Change:
} func updateISCSINode(b iscsiDiskMounter, tp string) error { <ins> // setting node.session.scan to manual to handle https://github.com/kubernetes/kubernetes/issues/90982 out, err := execWithLog(b, "iscsiadm", "-m", "node", "-p", tp, "-T", b.Iqn, "-I", b.Iface, "-o", "update", "-n", "node.session.scan", "-v", "manual") if err != nil { // don't fail if iscsiadm fails or the version does not support node.session.scan - log a warning to highlight the potential exposure klog.Warningf("iscsi: failed to update node with node.session.scan=manual, possible exposure to issue 90982: %v", out) } </ins> if !b.chapSession { return nil } <del> out, err := execWithLog(b, "iscsiadm", "-m", "node", "-p", tp, "-T", b.Iqn, "-I", b.Iface, "-o", "update", "-n", "node.session.auth.authmethod", "-v", "CHAP") </del> <ins> out, err = execWithLog(b, "iscsiadm", "-m", "node", "-p", tp, "-T", b.Iqn, "-I", b.Iface, "-o", "update", "-n", "node.session.auth.authmethod", "-v", "CHAP") </ins> if err != nil { return fmt.Errorf("iscsi: failed to update node with CHAP, output: %v", out) }

Discussion:
What happened: The issue stems from use cases were 1 initiator is used for multiple LUNs. Because of this, all of the LUNs are discovered and mapped on a node when it does its initial discovery / login via AttachDisk[1]. Because a pod never mounts the device on any of the other nodes, it is never removed. [1] In situations where LUN numbers can be reused, it’s possible to hit the below race condition where a new LUN reattaches to a faulty multipath device created for a previous LUN of the same number. What you expected to happen: Only the required LUN should be discovered and mounted. How to reproduce it (as minimally and precisely as possible): Assume we start from an empty environment using Node A and B. An empty environment is not a requirement, it just makes it visually easier. 1) User A creates a PVC, the PV and LUN1 are dynamically provisioned and mounted by the pod on Node A. Node A does an iscsiadm login (in addition to other things) and scans for the scsi devices, adds them to the multipath devicemap and attaches to /dev/dm-XX. 2) User B creates a second PVC, the PV and LUN2 are dynamically provisioned and mounted by the pod on Node B. Node B does an iscsiadm login (in addition to other things) and scans for the scsi devices, adds them to the multipath devicemap and attaches to a /dev/dm-XX device. This process finds both LUN1 and LUN2 and creates multipath devicemaps and devices for both of them. 3) User A creates a PVC, the PV and LUN3 are dynamically provisioned and mounted by the pod on Node A. Node A is already logged in because of LUN1 so it just scans for the new LUN, adds it to a multipath devicemap and attaches to a /dev/dm-XX device.

At this point: Node A knows about LUN1 and has it mounted and LUN3 and has it mounted Node B knows about LUN1 and LUN2 but only has LUN2 mounted 4) User A deletes their first pod and PVC from Node A. This triggers an unmount and the flushing and removal of the multipath device and underlying block devices on Node A. This deletes LUN1 on the backend. Because the pod was only running on Node A, the unmount and removal procedures do nothing for the discovered LUN1 and its devices on Node B. The multipath device for LUN1 on Node B loses its paths and goes into a faulty state. 5) User C creates a PVC, the PV and LUN1 (reused number) are dynamically provisioned and mounted by the pod on Node B. Node B is already logged in because of LUN2 and it already knows about LUN1 as it was discovered when LUN2 came along. In this case, LUN1 is reattached to the old multipath device and block devices from the previously deleted LUN1. This is when we see the LUN assignment / remap warning: Ex: May 5 10:02:58 server kernel: sd 40:0:0:29: Warning! Received an indication that the LUN assignments on this target have changed. The Linux SCSI layer does not automatical 6) Kubernetes tries to mount the multipath device to its pod and, based on our testing, sees a variety of possible issues, depending on what kind of leftover data is visible. These range from fixable fsck file system issues, fsck failures and complete disk formatting failures. The other primary issue we see is where there are PVC size mismatches and non fatal fsck fixes. If LUN1 was originally 1GB, the multipath and block devices reflect that. If the new PVC asked for 5GB, LUN1 is 5GB, but when it reconnects, the user only sees the previous 1GB devices. The reverse is also true where you could request 1GB and end up with the old 5GB block devices which results in formatting errors. Anything else we need to know?: The race condition is somewhat uncommon and depends on LUN number reuse and WWID stings.

As the number of LUNs grow and as more servers are restarted for patching etc.. there will be a great and greater chance of people hitting this as churn occurs. The other big issue which is independent of hitting the above race condition is the performance impact and stability of the server. Having a few hundred extra devices is not ideal, but the bigger issue looks to be the constant failed i/o’s and multipathd path down checks for LUNs which have been removed and remapped. Environment: kubernetes Kubernetes version (use ): master Cloud provider or hardware configuration: all OS (e.g: ): all Kernel (e.g. ): all
/sig storage
/assign