Change:
<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see ): No What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): iscsi session logout Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): Environment: Cloud provider or hardware configuration: Libvirt qemu VM, 4 CPU, 8Gig RAM OS (e.g. from /etc/os-release): Ubuntu 16.04.1 LTS (Xenial Xerus) Kernel (e.g. ): Linux localhost 4.4.0-62-generic -Ubuntu SMP Wed Jan 18 14:10:15 UTC 2017 x8664 x8664 x86_64 GNU/Linux Install tools: kubeadm Others: What happened: If you have multiple pods on the same node using PVCs that are backed by iSCSI LUNs on the same external system over a single iSCSI session (same IQN but different LUN numbers), then deleting one of the pods causes the iSCSI session to terminate for all pods on that node that are using it. For example, here and are nginx pods each connected to a different PVC, which are backed by LUNs residing on the same NetApp storage system. These share a single iSCSI session, differing only by LUN number. Once pod2 is deleted, all the connected LUNs are no longer available. [6:0:0:1] disk NETAPP LUN C-Mode 8300 /dev/sdc What you expected to happen: I expect that in-use iSCSI sessions are not terminated How to reproduce it (as minimally and precisely as possible): 2 PVs and PVCs that are backed by a single storage system that aggrates iSCSI connections into a single session. 2 pods that consume those PVCs one of the pods Anything else we need to know:
Could someone from please take a look? Thanks!
looking...
It looks like the problem is in line 86 of , in the function. The check evaluates whether a given mount path starts with the prefix (protocol + target portal + target IQN) for the LUN; this will always evaluate to false in most set-ups, since each path will look something like . Changing to appears to result in the correct behavior.
The problem is that the mount device shown in /proc/mounts are canonicalized, and thus when searching device prefix, iscsi volume failed to find a match. If we can use mount option, this the device will show up in /proc/mounts. But this is not a reliable fix since there are probable mpio paths too.
If I'm reading the code correctly, the function doesn't look at the actual device that's backing the mount point; it looks at the mount path (i.e., the second token in /proc/mounts, rather than the first). Thus, the change I proposed (HasPrefix -Contains) appears to work with some very basic testing. That said, I've not done anything with multipathing, so there may be something I'm missing; in addition, I suppose there's a chance that Contains may trigger false positives.
I see your point, yes that sounds right to me.
I think in this case, mpio device should work too (since the mount path constructed in the same way). I see you have yet another iscsi issue open, I am good with rescan. Can you create a pr to fix them?
I should be able create a PR for this issue; the LUN rescan PR will probably get handled by someone else on my team. Is the fix itself sufficient, or does it require tests?
thanks, the rescan fix doesn't have to unit test. The mount path fix needs also a unit test fix here

Discussion:
return refCount, nil } <del> // make a directory like /var/lib/kubelet/plugins/kubernetes.io/iscsi/portal-some_iqn-lun-lun_id func makePDNameInternal(host volume.VolumeHost, portal string, iqn string, lun string) string { return path.Join(host.GetPluginDir(iscsiPluginName), portal+"-"+iqn+"-lun-"+lun) </del> <ins> // make a directory like /var/lib/kubelet/plugins/kubernetes.io/iscsi/iface_name/portal-some_iqn-lun-lun_id func makePDNameInternal(host volume.VolumeHost, portal string, iqn string, lun string, iface string) string { return path.Join(host.GetPluginDir(iscsiPluginName), "iface-"+iface, portal+"-"+iqn+"-lun-"+lun) </ins> } type ISCSIUtil struct{} func (util *ISCSIUtil) MakeGlobalPDName(iscsi iscsiDisk) string { <del> return makePDNameInternal(iscsi.plugin.host, iscsi.portals[0], iscsi.iqn, iscsi.lun) </del> <ins> return makePDNameInternal(iscsi.plugin.host, iscsi.portals[0], iscsi.iqn, iscsi.lun, iscsi.iface) </ins> } func (util *ISCSIUtil) AttachDisk(b iscsiDiskMounter) error { var devicePath string var devicePaths []string var iscsiTransport string out, err := b.plugin.execCommand("iscsiadm", []string{"-m", "iface", "-I", b.iface, "-o", "show"}) if err != nil { glog.Errorf("iscsi: could not read iface %s error: %s", b.iface, string(out))

} iscsiTransport = extractTransportname(string(out)) bkpPortal := b.portals for _, tp := range bkpPortal { <ins> // Rescan sessions to discover newly mapped LUNs. Do not specify the interface when rescanning // to avoid establishing additional sessions to the same target. out, err := b.plugin.execCommand("iscsiadm", []string{"-m", "node", "-p", tp, "-T", b.iqn, "-R"}) if err != nil { glog.Errorf("iscsi: failed to rescan session with error: %s (%v)", string(out), err) } </ins> if iscsiTransport == "" { glog.Errorf("iscsi: could not find transport name in iface %s", b.iface) return errors.New(fmt.Sprintf("Could not parse iface file for %s", b.iface))

return err } refCount, err := getDevicePrefixRefCount(c.mounter, prefix) if err == nil && refCount == 0 { <del> // this portal/iqn are no longer referenced, log out // extract portal and iqn from device path </del> <ins> // This portal/iqn/iface is no longer referenced, log out. // Extract the portal and iqn from device path. </ins> portal, iqn, err := extractPortalAndIqn(device) if err != nil { return err } <del> glog.Infof("iscsi: log out target %s iqn %s", portal, iqn) out, err := c.plugin.execCommand("iscsiadm", []string{"-m", "node", "-p", portal, "-T", iqn, "--logout"}) if err != nil { glog.Errorf("iscsi: failed to detach disk Error: %s", string(out)) </del> <ins> // Extract the iface from the mountPath and use it to log out. If the iface // is not found, maintain the previous behavior to facilitate kubelet upgrade. // Logout may fail as no session may exist for the portal/IQN on the specified interface. iface, found := extractIface(mntPath) if found { glog.Infof("iscsi: log out target %s iqn %s iface %s", portal, iqn, iface) out, err := c.plugin.execCommand("iscsiadm", []string{"-m", "node", "-p", portal, "-T", iqn, "-I", iface, "--logout"}) if err != nil { glog.Errorf("iscsi: failed to detach disk Error: %s", string(out)) } } else { glog.Infof("iscsi: log out target %s iqn %s", portal, iqn) out, err := c.plugin.execCommand("iscsiadm", []string{"-m", "node", "-p", portal, "-T", iqn, "--logout"}) if err != nil { glog.Errorf("iscsi: failed to detach disk Error: %s", string(out)) } </ins> } } }

return "", "", fmt.Errorf("iscsi detach disk: malformatted mnt path: %s", mntPath) } device := mntPath[(ind + 1):] <del> // strip -lun- from device path ind = strings.LastIndex(device, "-lun-") </del> <ins> // strip -lun- from mount path ind = strings.LastIndex(mntPath, "-lun-") </ins> if ind < 0 { return "", "", fmt.Errorf("iscsi detach disk: malformatted mnt path: %s", mntPath) } <del> prefix := device[:ind] </del> <ins> prefix := mntPath[:ind] </ins> return device, prefix, nil } <ins> func extractIface(mntPath string) (string, bool) { re := regexp.MustCompile(`.+/iface-([^/]+)/.+`) re_output := re.FindStringSubmatch(mntPath) if re_output != nil { return re_output[1], true } return "", false } </ins> func extractPortalAndIqn(device string) (string, string, error) { ind1 := strings.Index(device, "-") if ind1 < 0 {

func TestExtractDeviceAndPrefix(t *testing.T) { devicePath := "127.0.0.1:3260-iqn.2014-12.com.example:test.tgt00" <ins> mountPrefix := "/var/lib/kubelet/plugins/kubernetes.io/iscsi/iface-default/" + devicePath </ins> lun := "-lun-0" <del> device, prefix, err := extractDeviceAndPrefix("/var/lib/kubelet/plugins/kubernetes.io/iscsi/" + devicePath + lun) if err != nil || device != (devicePath+lun) || prefix != devicePath { t.Errorf("extractDeviceAndPrefix: expected %s and %s, got %v %s and %s", devicePath+lun, devicePath, err, device, prefix) </del> <ins> device, prefix, err := extractDeviceAndPrefix(mountPrefix + lun) if err != nil || device != (devicePath+lun) || prefix != mountPrefix { t.Errorf("extractDeviceAndPrefix: expected %s and %s, got %v %s and %s", devicePath+lun, mountPrefix, err, device, prefix) } } func TestExtractIface(t *testing.T) { ifaceName := "default" devicePath := "127.0.0.1:3260-iqn.2014-12.com.example:test.tgt00-lun-0" iface, found := extractIface("/var/lib/kubelet/plugins/kubernetes.io/iscsi/iface-" + ifaceName + "/" + devicePath) if !found || iface != ifaceName { t.Errorf("extractIface: expected %s and %t, got %s and %t", ifaceName, true, iface, found) } iface, found = extractIface("/var/lib/kubelet/plugins/kubernetes.io/iscsi/" + devicePath) if found || iface != "" { t.Errorf("extractIface: expected %s and %t, got %s and %t", "", false, iface, found) </ins> } }